[fix](index) should not use light index change for bloom filter index by cambyzju · Pull Request #35214 · apache/doris

cambyzju · 2024-05-22T08:56:58Z

Proposed changes

Only inverted index support light index change, bitmap and bloomfilter index do not support light index change.

Before this pr, if we add or drop bitmap and bloomfilter index, it only do linked schema change, do not rewrite index data for history data.

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

doris-robot · 2024-05-22T08:57:04Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

cambyzju · 2024-05-22T08:58:13Z

run buildall

github-actions · 2024-05-22T09:02:50Z

clang-tidy review says "All clean, LGTM! 👍"

doris-robot · 2024-05-22T10:19:19Z

TeamCity be ut coverage result:
Function Coverage: 35.66% (9014/25279)
Line Coverage: 27.30% (74511/272976)
Region Coverage: 26.53% (38564/145375)
Branch Coverage: 23.37% (19663/84128)
Coverage Report: http://coverage.selectdb-in.cc/coverage/50828845400e5bb7e85d8ecd04a8805ddcb4e4d7_50828845400e5bb7e85d8ecd04a8805ddcb4e4d7/report/index.html

doris-robot · 2024-05-22T10:24:24Z

TPC-DS: Total hot run time: 170468 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 50828845400e5bb7e85d8ecd04a8805ddcb4e4d7, data reload: false
query1	915	383	373	373
query2	6289	2339	2346	2339
query3	6629	205	205	205
query4	18989	17853	17765	17765
query5	3887	429	412	412
query6	235	177	151	151
query7	4690	322	298	298
query8	254	198	192	192
query9	8798	2483	2435	2435
query10	421	278	276	276
query11	10975	10562	10616	10562
query12	119	89	94	89
query13	1647	351	368	351
query14	9606	7821	7586	7586
query15	245	179	182	179
query16	8339	266	260	260
query17	1766	536	524	524
query18	1929	260	254	254
query19	194	147	153	147
query20	89	78	78	78
query21	189	123	124	123
query22	4324	4004	3868	3868
query23	33610	32871	32972	32871
query24	11118	2817	2753	2753
query25	586	348	344	344
query26	1329	161	149	149
query27	2958	303	301	301
query28	7562	2067	2029	2029
query29	880	623	598	598
query30	255	164	179	164
query31	979	733	742	733
query32	89	49	51	49
query33	758	251	265	251
query34	982	454	474	454
query35	734	586	599	586
query36	1069	895	916	895
query37	135	70	69	69
query38	2910	2771	2747	2747
query39	832	801	791	791
query40	210	123	123	123
query41	44	42	41	41
query42	102	99	98	98
query43	576	524	526	524
query44	1221	740	732	732
query45	175	161	162	161
query46	1076	720	702	702
query47	1860	1759	1773	1759
query48	352	298	287	287
query49	881	375	410	375
query50	746	391	382	382
query51	6955	6772	6826	6772
query52	102	94	93	93
query53	356	287	279	279
query54	903	415	415	415
query55	71	69	73	69
query56	253	240	241	240
query57	1149	1015	1047	1015
query58	226	197	203	197
query59	3258	3235	3120	3120
query60	276	248	250	248
query61	87	85	85	85
query62	676	460	459	459
query63	310	278	289	278
query64	9595	2172	1690	1690
query65	3171	3089	3105	3089
query66	1348	325	318	318
query67	15371	14816	14923	14816
query68	4684	520	525	520
query69	499	268	284	268
query70	1118	1020	1132	1020
query71	458	272	327	272
query72	7407	5405	2698	2698
query73	710	321	317	317
query74	6052	5620	5518	5518
query75	3648	2609	2611	2609
query76	3454	1040	1026	1026
query77	570	264	262	262
query78	10216	9804	9957	9804
query79	2022	526	514	514
query80	823	433	433	433
query81	528	239	253	239
query82	1206	92	95	92
query83	220	169	162	162
query84	231	84	81	81
query85	1489	300	345	300
query86	469	317	285	285
query87	3306	3106	3156	3106
query88	4157	2342	2348	2342
query89	479	370	370	370
query90	1976	189	182	182
query91	121	107	95	95
query92	62	45	47	45
query93	1868	485	490	485
query94	1141	181	180	180
query95	393	297	308	297
query96	600	263	262	262
query97	3173	2995	3044	2995
query98	237	216	211	211
query99	1088	860	891	860
Total cold run time: 273070 ms
Total hot run time: 170468 ms

doris-robot · 2024-05-22T10:29:57Z

ClickBench: Total hot run time: 31.07 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 50828845400e5bb7e85d8ecd04a8805ddcb4e4d7, data reload: false
query1	0.04	0.03	0.04
query2	0.09	0.04	0.04
query3	0.24	0.05	0.04
query4	1.67	0.08	0.08
query5	0.50	0.48	0.51
query6	1.13	0.72	0.73
query7	0.02	0.01	0.02
query8	0.05	0.05	0.04
query9	0.54	0.47	0.47
query10	0.52	0.54	0.55
query11	0.16	0.12	0.11
query12	0.15	0.12	0.13
query13	0.59	0.59	0.60
query14	0.76	0.78	0.77
query15	0.83	0.81	0.81
query16	0.36	0.37	0.36
query17	0.98	0.94	1.00
query18	0.22	0.25	0.24
query19	1.79	1.68	1.70
query20	0.01	0.01	0.01
query21	15.54	0.68	0.66
query22	3.65	7.56	2.30
query23	18.25	1.49	1.34
query24	1.55	0.23	0.25
query25	0.15	0.08	0.09
query26	0.25	0.16	0.17
query27	0.08	0.08	0.07
query28	13.58	1.01	0.99
query29	13.79	3.36	3.31
query30	0.25	0.06	0.05
query31	2.84	0.40	0.39
query32	3.25	0.48	0.47
query33	2.92	2.93	2.93
query34	17.25	4.44	4.49
query35	4.48	4.52	4.67
query36	0.65	0.46	0.46
query37	0.18	0.15	0.15
query38	0.15	0.14	0.14
query39	0.04	0.04	0.03
query40	0.16	0.14	0.14
query41	0.09	0.05	0.05
query42	0.05	0.04	0.05
query43	0.04	0.04	0.04
Total cold run time: 109.84 s
Total hot run time: 31.07 s

github-actions · 2024-05-22T11:14:34Z

clang-tidy review says "All clean, LGTM! 👍"

cambyzju · 2024-05-23T02:34:27Z

run buildall

lide-reed

LGTM

github-actions · 2024-05-23T02:51:03Z

PR approved by at least one committer and no changes requested.

github-actions · 2024-05-23T02:51:05Z

PR approved by anyone and no changes requested.

doris-robot · 2024-05-23T03:42:45Z

TPC-H: Total hot run time: 42533 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit e3a0f147befa0cb249dc83ca220283f85d112b40, data reload: false
------ Round 1 ----------------------------------
q1	17639	5041	4339	4339
q2	2013	185	206	185
q3	10455	1277	1192	1192
q4	10190	879	820	820
q5	7500	2798	2756	2756
q6	225	133	137	133
q7	969	607	603	603
q8	9211	2174	2172	2172
q9	9518	6851	6766	6766
q10	9336	3999	3899	3899
q11	444	260	232	232
q12	425	230	241	230
q13	17377	3195	3291	3195
q14	266	212	232	212
q15	520	474	486	474
q16	513	403	389	389
q17	1000	807	725	725
q18	8474	7979	7909	7909
q19	6146	1584	1586	1584
q20	642	319	317	317
q21	5257	4104	4170	4104
q22	356	305	297	297
Total cold run time: 118476 ms
Total hot run time: 42533 ms
----- Round 2, with runtime_filter_mode=off -----
q1	4624	4479	4415	4415
q2	386	296	282	282
q3	3160	2977	2872	2872
q4	1933	1669	1648	1648
q5	5511	5510	5527	5510
q6	224	125	126	125
q7	2181	1841	1832	1832
q8	3308	3451	3453	3451
q9	8677	8735	8791	8735
q10	3986	3750	3869	3750
q11	598	503	492	492
q12	836	647	634	634
q13	15999	3160	3129	3129
q14	308	296	285	285
q15	561	491	483	483
q16	512	430	442	430
q17	1829	1560	1524	1524
q18	7923	7672	7603	7603
q19	1712	1623	1555	1555
q20	2009	1807	1815	1807
q21	9077	4853	4709	4709
q22	559	520	475	475
Total cold run time: 75913 ms
Total hot run time: 55746 ms

doris-robot · 2024-05-23T03:46:31Z

TeamCity be ut coverage result:
Function Coverage: 35.65% (9012/25279)
Line Coverage: 27.29% (74514/272999)
Region Coverage: 26.53% (38572/145395)
Branch Coverage: 23.38% (19672/84144)
Coverage Report: http://coverage.selectdb-in.cc/coverage/e3a0f147befa0cb249dc83ca220283f85d112b40_e3a0f147befa0cb249dc83ca220283f85d112b40/report/index.html

doris-robot · 2024-05-23T03:53:18Z

TPC-DS: Total hot run time: 173391 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit e3a0f147befa0cb249dc83ca220283f85d112b40, data reload: false
query1	924	375	375	375
query2	6445	2528	2349	2349
query3	6639	212	211	211
query4	19294	17473	17457	17457
query5	4083	420	420	420
query6	236	163	151	151
query7	4585	300	293	293
query8	241	185	179	179
query9	8673	2370	2359	2359
query10	456	283	293	283
query11	10444	10199	10134	10134
query12	139	94	89	89
query13	1637	368	370	368
query14	9906	7705	7422	7422
query15	215	175	173	173
query16	7568	261	268	261
query17	1307	538	529	529
query18	1928	283	272	272
query19	197	152	151	151
query20	94	84	85	84
query21	197	128	127	127
query22	4335	4001	4004	4001
query23	33430	33055	33177	33055
query24	5309	2860	2957	2860
query25	471	356	369	356
query26	698	156	157	156
query27	1844	319	325	319
query28	3618	2058	2089	2058
query29	862	606	611	606
query30	245	173	176	173
query31	910	756	766	756
query32	61	53	53	53
query33	495	261	260	260
query34	841	481	474	474
query35	721	606	595	595
query36	1047	905	894	894
query37	104	72	77	72
query38	2902	2824	2761	2761
query39	845	784	797	784
query40	198	123	122	122
query41	47	43	43	43
query42	102	98	94	94
query43	603	545	530	530
query44	1050	716	733	716
query45	174	159	163	159
query46	1053	709	737	709
query47	1873	1772	1750	1750
query48	376	294	290	290
query49	757	378	427	378
query50	771	390	382	382
query51	6899	6606	6689	6606
query52	100	88	88	88
query53	352	291	276	276
query54	535	420	426	420
query55	74	72	74	72
query56	277	240	248	240
query57	1122	1049	1032	1032
query58	236	202	226	202
query59	3494	3203	3093	3093
query60	282	247	250	247
query61	91	90	88	88
query62	548	456	445	445
query63	315	283	281	281
query64	2627	1714	1698	1698
query65	3230	3112	3144	3112
query66	813	326	324	324
query67	15192	15163	14838	14838
query68	4558	537	541	537
query69	437	266	269	266
query70	1105	1131	1106	1106
query71	384	269	258	258
query72	7336	5585	5793	5585
query73	726	322	321	321
query74	5949	5778	5775	5775
query75	3269	2615	2638	2615
query76	2269	982	1026	982
query77	366	261	261	261
query78	10411	9934	9866	9866
query79	2165	518	509	509
query80	879	441	428	428
query81	515	248	244	244
query82	761	95	94	94
query83	259	171	191	171
query84	247	89	85	85
query85	1148	275	275	275
query86	461	305	309	305
query87	3299	3144	3158	3144
query88	3931	2342	2357	2342
query89	481	394	391	391
query90	2066	196	191	191
query91	132	107	106	106
query92	59	51	56	51
query93	2557	513	492	492
query94	1246	192	192	192
query95	414	316	314	314
query96	597	267	266	266
query97	3212	3002	3020	3002
query98	247	218	228	218
query99	1268	967	844	844
Total cold run time: 249891 ms
Total hot run time: 173391 ms

doris-robot · 2024-05-23T03:58:48Z

ClickBench: Total hot run time: 30.97 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit e3a0f147befa0cb249dc83ca220283f85d112b40, data reload: false
query1	0.04	0.04	0.03
query2	0.08	0.04	0.05
query3	0.23	0.04	0.05
query4	1.69	0.09	0.10
query5	0.55	0.51	0.52
query6	1.12	0.73	0.72
query7	0.02	0.01	0.02
query8	0.05	0.04	0.04
query9	0.53	0.49	0.49
query10	0.55	0.53	0.53
query11	0.16	0.12	0.12
query12	0.15	0.12	0.12
query13	0.59	0.60	0.61
query14	0.76	0.78	0.79
query15	0.84	0.82	0.80
query16	0.35	0.38	0.36
query17	0.94	0.99	1.01
query18	0.22	0.23	0.24
query19	1.74	1.71	1.69
query20	0.01	0.02	0.01
query21	15.44	0.74	0.69
query22	4.51	6.85	2.08
query23	18.28	1.33	1.34
query24	1.86	0.25	0.20
query25	0.15	0.08	0.08
query26	0.27	0.18	0.16
query27	0.08	0.08	0.08
query28	13.26	1.03	1.01
query29	13.33	3.33	3.31
query30	0.24	0.06	0.05
query31	2.87	0.38	0.39
query32	3.27	0.47	0.47
query33	2.93	2.91	2.90
query34	17.03	4.47	4.47
query35	4.59	4.51	4.65
query36	0.66	0.47	0.46
query37	0.18	0.16	0.15
query38	0.15	0.15	0.14
query39	0.04	0.04	0.04
query40	0.16	0.14	0.14
query41	0.09	0.05	0.04
query42	0.06	0.04	0.05
query43	0.04	0.04	0.04
Total cold run time: 110.11 s
Total hot run time: 30.97 s

xy720

LGTM

xiaokang

LGTM

…35214 (#34732)

…#35214) ## Proposed changes Only inverted index support light index change, bitmap and bloomfilter index do not support light index change. Before this pr, if we add or drop bitmap and bloomfilter index, it only do linked schema change, do not rewrite index data for history data.

…pache#35214 (apache#34732)

…apache#35214) ## Proposed changes Only inverted index support light index change, bitmap and bloomfilter index do not support light index change. Before this pr, if we add or drop bitmap and bloomfilter index, it only do linked schema change, do not rewrite index data for history data.

cambyzju requested a review from dataroaring May 22, 2024 08:57

cambyzju added dev/2.0.x dev/2.1.x labels May 22, 2024

cambyzju mentioned this pull request May 22, 2024
[fix](index) bitmap and bloomfilter index should not do light index change #35225
Merged

bugfix for add and drop index for history data
e3a0f14

cambyzju force-pushed the fix-master-0521 branch from 5082884 to e3a0f14Compare May 22, 2024 11:08

cambyzju mentioned this pull request May 22, 2024
[fix](index) should not use light index change for bloom filter index #34732
Merged

lide-reed approved these changes May 23, 2024
View reviewed changes

github-actionsBot added the approved Indicates a PR has been approved by one committer. label May 23, 2024

github-actionsBot added the reviewed label May 23, 2024

xy720 approved these changes May 28, 2024
View reviewed changes

xy720 merged commit f7552f4 into apache:masterMay 28, 2024

xy720 added area/bloom-filter-index Issues or PRS related to bloom filter index area/bitmap-index labels May 28, 2024

xiaokang reviewed May 28, 2024
View reviewed changes

xiaokang pushed a commit that referenced this pull request May 28, 2024
[fix](index) should not use light index change for bloom filter index#…
b511413
…35214 (#34732)

xiaokang added usercase Important user case type label dev/2.0.11-merged and removed dev/2.0.x labels May 28, 2024

dataroaring added dev/2.1.4-merged and removed dev/2.1.x labels Jun 6, 2024

dataroaring added the dev/3.0.0-merged label Aug 5, 2024

mongo360 pushed a commit to mongo360/doris that referenced this pull request Aug 16, 2024
[fix](index) should not use light index change for bloom filter indexa…
b9bb6d4
…pache#35214 (apache#34732)

Uh oh!

Conversation

cambyzju commented May 22, 2024• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Proposed changes

Further comments

Uh oh!

doris-robot commented May 22, 2024

Uh oh!

cambyzju commented May 22, 2024

Uh oh!

github-actionsBot commented May 22, 2024

Uh oh!

doris-robot commented May 22, 2024

Uh oh!

doris-robot commented May 22, 2024

Uh oh!

doris-robot commented May 22, 2024

Uh oh!

github-actionsBot commented May 22, 2024

Uh oh!

cambyzju commented May 23, 2024

Uh oh!

lide-reed left a comment

Choose a reason for hiding this comment

Uh oh!

github-actionsBot commented May 23, 2024

Uh oh!

github-actionsBot commented May 23, 2024

Uh oh!

doris-robot commented May 23, 2024

Uh oh!

doris-robot commented May 23, 2024

Uh oh!

doris-robot commented May 23, 2024

Uh oh!

doris-robot commented May 23, 2024

Uh oh!

xy720 left a comment

Choose a reason for hiding this comment

Uh oh!

xiaokang left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

cambyzju commented May 22, 2024•
edited
Loading