[Refactor](inverted index) refactor inverted index interface in tablet schema by csun5285 · Pull Request #43003 · apache/doris

csun5285 · 2024-10-31T08:57:17Z

Proposed changes

refactor inverted index interface in tablet schema

There's a large amount of similar code

for (const auto& index : schema_ptr->indexes()) {
if (index.index_type() == IndexType::INVERTED) {
....
}
}

Now it has been changed to

for (const auto& index : schema_ptr->inverted_indexes()) {
....
}

Remove functions with redundant functionality, such as get_indexes_for_column , get_inverted_index_with_index_id ....
Retrieve the index meta for the inverted index. Currently, there are two types: one is inverted_index(TabletColumn), which checks the column type (e.g., if the column type is float, it does not support indexing and will be filtered out); the other is inverted_index(unique_id, index_suffix), which retrieves the index meta for the current column without checking the column type.

doris-robot · 2024-10-31T08:57:22Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

csun5285 · 2024-10-31T08:57:30Z

run buildall

github-actions

clang-tidy made some suggestions

doris-robot · 2024-10-31T10:10:21Z

TPC-H: Total hot run time: 41471 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 9f393cd3f646285b5411727b3bd70a30611df9cc, data reload: false
------ Round 1 ----------------------------------
q1	17571	7492	7328	7328
q2	2044	161	156	156
q3	10565	1149	1116	1116
q4	10224	884	821	821
q5	7758	3108	3120	3108
q6	234	145	146	145
q7	1018	605	607	605
q8	9362	1965	2064	1965
q9	6684	6460	6458	6458
q10	7059	2443	2416	2416
q11	443	238	238	238
q12	412	216	211	211
q13	17763	2968	3026	2968
q14	242	210	205	205
q15	584	533	516	516
q16	625	581	598	581
q17	984	544	687	544
q18	7446	6664	6836	6664
q19	1338	1000	1070	1000
q20	474	183	181	181
q21	4092	3222	3249	3222
q22	1131	1044	1023	1023
Total cold run time: 108053 ms
Total hot run time: 41471 ms
----- Round 2, with runtime_filter_mode=off -----
q1	7253	7260	7244	7244
q2	324	223	228	223
q3	2930	2784	2869	2784
q4	1962	1731	1756	1731
q5	5487	5530	5556	5530
q6	217	137	139	137
q7	2210	1717	1698	1698
q8	3260	3420	3439	3420
q9	8681	8602	8657	8602
q10	3528	3483	3493	3483
q11	574	476	484	476
q12	794	567	587	567
q13	9047	3027	3037	3027
q14	295	266	259	259
q15	565	516	501	501
q16	667	618	621	618
q17	1853	1615	1567	1567
q18	7938	7505	7552	7505
q19	1682	1574	1540	1540
q20	2064	1807	1803	1803
q21	5426	5221	5279	5221
q22	1111	1041	1016	1016
Total cold run time: 67868 ms
Total hot run time: 58952 ms

doris-robot · 2024-10-31T10:19:23Z

TeamCity be ut coverage result:
Function Coverage: 37.94% (9848/25957)
Line Coverage: 29.20% (82070/281044)
Region Coverage: 28.45% (42333/148789)
Branch Coverage: 25.03% (21507/85930)
Coverage Report: http://coverage.selectdb-in.cc/coverage/9f393cd3f646285b5411727b3bd70a30611df9cc_9f393cd3f646285b5411727b3bd70a30611df9cc/report/index.html

doris-robot · 2024-10-31T10:22:28Z

TPC-DS: Total hot run time: 191990 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 9f393cd3f646285b5411727b3bd70a30611df9cc, data reload: false
query1	980	366	367	366
query2	6513	2078	2077	2077
query3	6788	223	214	214
query4	34148	23452	23595	23452
query5	4299	427	437	427
query6	260	166	166	166
query7	4602	294	298	294
query8	286	220	220	220
query9	9691	2713	2693	2693
query10	464	230	237	230
query11	18351	15416	15136	15136
query12	145	106	100	100
query13	1658	410	424	410
query14	10339	7390	7406	7390
query15	300	174	177	174
query16	8132	460	437	437
query17	1770	572	552	552
query18	2140	292	286	286
query19	355	160	146	146
query20	116	110	107	107
query21	209	101	99	99
query22	4381	4343	4437	4343
query23	34981	34135	34359	34135
query24	11122	2757	2836	2757
query25	683	396	394	394
query26	1236	161	157	157
query27	2718	281	286	281
query28	8037	2470	2484	2470
query29	878	415	409	409
query30	319	153	158	153
query31	1020	820	820	820
query32	90	55	57	55
query33	763	272	264	264
query34	1004	514	544	514
query35	880	724	745	724
query36	1083	952	964	952
query37	128	76	69	69
query38	4503	4331	4279	4279
query39	1617	1403	1435	1403
query40	200	101	103	101
query41	48	45	45	45
query42	108	96	97	96
query43	534	483	490	483
query44	1340	839	822	822
query45	186	168	166	166
query46	1114	701	710	701
query47	1939	1837	1869	1837
query48	410	319	341	319
query49	1168	397	396	396
query50	800	386	397	386
query51	7288	7040	7019	7019
query52	98	91	88	88
query53	252	179	181	179
query54	1229	403	421	403
query55	81	80	77	77
query56	268	251	243	243
query57	1333	1210	1159	1159
query58	245	205	207	205
query59	3238	3069	3138	3069
query60	281	259	251	251
query61	122	115	120	115
query62	894	686	687	686
query63	220	193	210	193
query64	5222	613	584	584
query65	3302	3235	3214	3214
query66	1432	304	314	304
query67	16054	15669	15716	15669
query68	4837	555	547	547
query69	439	255	254	254
query70	1219	1132	1150	1132
query71	340	252	244	244
query72	6371	3975	3975	3975
query73	764	357	365	357
query74	10350	8954	9043	8954
query75	3409	2717	2665	2665
query76	2924	1095	982	982
query77	412	277	277	277
query78	10449	9435	9436	9435
query79	1834	599	597	597
query80	1145	411	417	411
query81	545	243	236	236
query82	953	119	119	119
query83	208	134	135	134
query84	231	67	75	67
query85	1356	284	282	282
query86	432	310	304	304
query87	4873	4733	4737	4733
query88	3335	2201	2164	2164
query89	392	287	288	287
query90	2030	186	187	186
query91	129	97	102	97
query92	66	50	48	48
query93	1760	561	563	561
query94	934	260	285	260
query95	342	240	243	240
query96	623	276	281	276
query97	2877	2699	2729	2699
query98	215	194	194	194
query99	1538	1308	1310	1308
Total cold run time: 303727 ms
Total hot run time: 191990 ms

doris-robot · 2024-10-31T10:27:57Z

ClickBench: Total hot run time: 33.06 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 9f393cd3f646285b5411727b3bd70a30611df9cc, data reload: false
query1	0.03	0.03	0.03
query2	0.07	0.03	0.04
query3	0.23	0.06	0.06
query4	1.65	0.10	0.10
query5	0.42	0.40	0.39
query6	1.18	0.65	0.64
query7	0.02	0.02	0.02
query8	0.04	0.03	0.03
query9	0.56	0.50	0.49
query10	0.55	0.54	0.56
query11	0.16	0.10	0.10
query12	0.14	0.11	0.11
query13	0.61	0.60	0.60
query14	2.71	2.76	2.76
query15	0.89	0.82	0.83
query16	0.38	0.38	0.39
query17	1.05	0.98	0.99
query18	0.24	0.23	0.21
query19	1.90	1.88	1.96
query20	0.01	0.02	0.01
query21	15.41	0.61	0.59
query22	2.60	2.81	1.72
query23	17.21	0.86	0.81
query24	2.99	2.09	1.56
query25	0.24	0.10	0.12
query26	0.54	0.14	0.14
query27	0.04	0.05	0.03
query28	9.25	1.09	1.07
query29	12.58	3.25	3.23
query30	0.25	0.06	0.06
query31	2.88	0.38	0.36
query32	3.29	0.47	0.45
query33	2.96	3.02	3.06
query34	16.93	4.46	4.45
query35	4.52	4.59	4.51
query36	0.67	0.49	0.48
query37	0.08	0.06	0.06
query38	0.04	0.04	0.03
query39	0.03	0.02	0.02
query40	0.16	0.12	0.12
query41	0.07	0.02	0.03
query42	0.03	0.02	0.02
query43	0.03	0.03	0.03
Total cold run time: 105.64 s
Total hot run time: 33.06 s

csun5285 · 2024-10-31T12:25:35Z

run buildall

csun5285 · 2024-10-31T12:29:09Z

run buildall

csun5285 · 2024-10-31T13:57:20Z

run buildall

doris-robot · 2024-10-31T15:13:44Z

TeamCity be ut coverage result:
Function Coverage: 37.81% (9819/25968)
Line Coverage: 28.99% (81582/281392)
Region Coverage: 28.28% (42129/148963)
Branch Coverage: 24.85% (21370/86002)
Coverage Report: http://coverage.selectdb-in.cc/coverage/70bca5849b6b44767e7222f4471cb43c2ff2772d_70bca5849b6b44767e7222f4471cb43c2ff2772d/report/index.html

doris-robot · 2024-11-01T01:51:02Z

TPC-H: Total hot run time: 41336 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 70bca5849b6b44767e7222f4471cb43c2ff2772d, data reload: false
------ Round 1 ----------------------------------
q1	17552	7793	7346	7346
q2	2052	180	160	160
q3	10596	1063	1206	1063
q4	10580	852	897	852
q5	7734	3136	3117	3117
q6	236	146	146	146
q7	1016	609	610	609
q8	9374	1962	2051	1962
q9	6684	6440	6495	6440
q10	7062	2462	2432	2432
q11	470	260	256	256
q12	407	220	215	215
q13	17777	3034	2990	2990
q14	238	210	209	209
q15	579	525	526	525
q16	661	559	597	559
q17	969	552	563	552
q18	7276	6777	6683	6683
q19	1336	1061	972	972
q20	495	180	178	178
q21	4014	3204	3093	3093
q22	1127	1018	977	977
Total cold run time: 108235 ms
Total hot run time: 41336 ms
----- Round 2, with runtime_filter_mode=off -----
q1	7325	7408	7276	7276
q2	332	229	225	225
q3	3052	2963	2951	2951
q4	2168	1872	1887	1872
q5	5749	5819	5847	5819
q6	228	137	139	137
q7	2208	1866	1778	1778
q8	3440	3519	3456	3456
q9	8944	9002	8937	8937
q10	3641	3572	3589	3572
q11	609	499	518	499
q12	850	645	639	639
q13	9120	3225	3240	3225
q14	297	285	291	285
q15	587	530	530	530
q16	684	640	662	640
q17	1852	1617	1605	1605
q18	8482	7784	7598	7598
q19	1717	1532	1639	1532
q20	2151	1867	1875	1867
q21	5670	5396	5457	5396
q22	1142	1066	1035	1035
Total cold run time: 70248 ms
Total hot run time: 60874 ms

doris-robot · 2024-11-01T02:03:35Z

TPC-DS: Total hot run time: 195526 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 70bca5849b6b44767e7222f4471cb43c2ff2772d, data reload: false
query1	1235	892	881	881
query2	6911	2152	2133	2133
query3	10939	4088	4130	4088
query4	68383	28821	23530	23530
query5	5005	453	439	439
query6	393	168	162	162
query7	5520	289	298	289
query8	334	228	234	228
query9	8712	2735	2733	2733
query10	418	253	245	245
query11	17190	15219	15856	15219
query12	164	102	106	102
query13	1527	440	429	429
query14	9975	7334	6799	6799
query15	218	193	188	188
query16	7071	478	457	457
query17	1026	551	561	551
query18	1770	298	290	290
query19	217	157	146	146
query20	121	110	111	110
query21	197	97	102	97
query22	4655	4350	4353	4350
query23	35048	34112	33823	33823
query24	5990	2749	2671	2671
query25	497	391	393	391
query26	650	158	156	156
query27	1657	291	296	291
query28	4059	2446	2444	2444
query29	683	423	416	416
query30	230	157	155	155
query31	1004	822	786	786
query32	61	58	66	58
query33	474	272	264	264
query34	915	509	537	509
query35	839	722	711	711
query36	1075	966	983	966
query37	121	73	74	73
query38	4325	4210	4284	4210
query39	1472	1452	1385	1385
query40	202	100	99	99
query41	48	47	46	46
query42	111	100	100	100
query43	545	487	486	486
query44	1177	831	828	828
query45	181	169	167	167
query46	1125	678	681	678
query47	1952	1849	1867	1849
query48	428	318	326	318
query49	727	408	409	408
query50	805	394	404	394
query51	7179	7205	7188	7188
query52	101	92	88	88
query53	254	181	179	179
query54	512	404	404	404
query55	74	76	79	76
query56	266	249	243	243
query57	1304	1156	1150	1150
query58	221	207	206	206
query59	3313	2946	3086	2946
query60	275	241	249	241
query61	99	97	106	97
query62	776	698	666	666
query63	228	194	186	186
query64	1322	624	585	585
query65	3268	3178	3215	3178
query66	635	295	297	295
query67	15920	15800	15643	15643
query68	3668	588	585	585
query69	433	252	253	252
query70	1194	1075	1140	1075
query71	327	250	261	250
query72	6268	3960	3988	3960
query73	755	354	366	354
query74	9879	8997	9051	8997
query75	3388	2653	2695	2653
query76	1807	1010	1114	1010
query77	496	300	266	266
query78	10491	9394	9356	9356
query79	1465	592	589	589
query80	830	436	422	422
query81	532	242	246	242
query82	770	119	116	116
query83	159	132	139	132
query84	275	66	69	66
query85	826	293	270	270
query86	331	305	297	297
query87	4807	4775	4742	4742
query88	3723	2206	2185	2185
query89	410	289	288	288
query90	1906	187	182	182
query91	133	99	100	99
query92	66	48	46	46
query93	1680	539	542	539
query94	724	281	289	281
query95	344	238	246	238
query96	609	287	297	287
query97	2900	2717	2731	2717
query98	226	205	193	193
query99	1600	1318	1340	1318
Total cold run time: 317151 ms
Total hot run time: 195526 ms

doris-robot · 2024-11-01T02:09:06Z

ClickBench: Total hot run time: 31.8 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 70bca5849b6b44767e7222f4471cb43c2ff2772d, data reload: false
query1	0.03	0.03	0.03
query2	0.07	0.03	0.03
query3	0.24	0.07	0.07
query4	1.63	0.10	0.10
query5	0.42	0.41	0.41
query6	1.15	0.66	0.64
query7	0.02	0.02	0.01
query8	0.05	0.04	0.03
query9	0.57	0.49	0.49
query10	0.55	0.54	0.57
query11	0.14	0.10	0.10
query12	0.15	0.11	0.11
query13	0.61	0.60	0.59
query14	2.72	2.81	2.84
query15	0.91	0.83	0.82
query16	0.36	0.37	0.38
query17	1.05	1.08	1.06
query18	0.20	0.19	0.20
query19	1.96	1.88	2.04
query20	0.01	0.01	0.01
query21	15.35	0.60	0.59
query22	2.46	2.98	1.44
query23	17.11	0.96	0.76
query24	2.63	0.49	2.28
query25	0.36	0.08	0.07
query26	0.44	0.13	0.13
query27	0.05	0.05	0.04
query28	10.41	1.09	1.09
query29	12.54	3.23	3.22
query30	0.25	0.06	0.06
query31	2.86	0.38	0.38
query32	3.29	0.47	0.46
query33	2.96	3.02	3.03
query34	17.06	4.52	4.51
query35	4.54	4.48	4.55
query36	0.67	0.48	0.47
query37	0.08	0.06	0.05
query38	0.05	0.03	0.04
query39	0.03	0.03	0.02
query40	0.16	0.13	0.13
query41	0.07	0.03	0.02
query42	0.03	0.02	0.02
query43	0.04	0.03	0.04
Total cold run time: 106.28 s
Total hot run time: 31.8 s

…t schema

csun5285 · 2024-11-04T05:00:18Z

run buildall

doris-robot · 2024-11-04T06:37:55Z

TeamCity be ut coverage result:
Function Coverage: 37.82% (9821/25968)
Line Coverage: 28.99% (81612/281530)
Region Coverage: 28.26% (42124/149078)
Branch Coverage: 24.84% (21375/86052)
Coverage Report: http://coverage.selectdb-in.cc/coverage/b6e22ec5cb82a9ae27d793046c7f474b01e2f833_b6e22ec5cb82a9ae27d793046c7f474b01e2f833/report/index.html

doris-robot · 2024-11-04T06:39:09Z

TPC-H: Total hot run time: 41409 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit b6e22ec5cb82a9ae27d793046c7f474b01e2f833, data reload: false
------ Round 1 ----------------------------------
q1	17616	7641	7351	7351
q2	2074	168	180	168
q3	10671	1103	1180	1103
q4	10406	918	794	794
q5	7766	3119	3095	3095
q6	237	149	145	145
q7	1020	609	602	602
q8	9348	1967	2036	1967
q9	6612	6470	6464	6464
q10	7081	2419	2451	2419
q11	466	257	266	257
q12	415	223	207	207
q13	17798	3036	2999	2999
q14	236	210	218	210
q15	580	515	518	515
q16	668	609	589	589
q17	1008	571	586	571
q18	7458	6710	6568	6568
q19	1327	1043	1062	1043
q20	478	181	180	180
q21	4014	3160	3193	3160
q22	1115	1016	1002	1002
Total cold run time: 108394 ms
Total hot run time: 41409 ms
----- Round 2, with runtime_filter_mode=off -----
q1	7260	7240	7271	7240
q2	328	222	222	222
q3	3043	2926	2943	2926
q4	2125	1909	1777	1777
q5	5837	5823	5854	5823
q6	235	138	141	138
q7	2296	1877	1803	1803
q8	3427	3560	3432	3432
q9	8955	8989	8973	8973
q10	3586	3570	3587	3570
q11	613	507	501	501
q12	844	622	590	590
q13	11076	3233	3182	3182
q14	307	265	285	265
q15	580	517	528	517
q16	685	623	662	623
q17	1840	1637	1613	1613
q18	8385	7967	7808	7808
q19	1736	1586	1651	1586
q20	2126	1872	1863	1863
q21	5688	5367	5534	5367
q22	1111	1046	1066	1046
Total cold run time: 72083 ms
Total hot run time: 60865 ms

doris-robot · 2024-11-04T06:51:47Z

TPC-DS: Total hot run time: 195949 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit b6e22ec5cb82a9ae27d793046c7f474b01e2f833, data reload: false
query1	1230	939	919	919
query2	6227	2077	2083	2077
query3	10818	4064	4045	4045
query4	67887	29856	23605	23605
query5	4916	448	435	435
query6	404	167	160	160
query7	5613	292	306	292
query8	313	235	218	218
query9	9298	2752	2749	2749
query10	465	256	248	248
query11	17273	15308	15720	15308
query12	156	108	107	107
query13	1586	472	449	449
query14	10423	7516	7096	7096
query15	209	186	175	175
query16	7076	464	459	459
query17	1011	556	561	556
query18	1809	303	295	295
query19	192	152	147	147
query20	119	110	112	110
query21	203	101	104	101
query22	4848	4320	4483	4320
query23	34354	33941	33881	33881
query24	6039	2761	2686	2686
query25	507	387	404	387
query26	649	156	154	154
query27	1703	280	289	280
query28	3973	2493	2451	2451
query29	678	431	426	426
query30	230	152	148	148
query31	991	803	822	803
query32	67	55	56	55
query33	400	269	272	269
query34	901	502	506	502
query35	844	737	710	710
query36	1064	916	939	916
query37	118	72	77	72
query38	4380	4201	4307	4201
query39	1482	1439	1440	1439
query40	204	101	99	99
query41	48	46	47	46
query42	110	102	98	98
query43	536	482	491	482
query44	1179	830	846	830
query45	181	166	167	166
query46	1110	711	678	678
query47	1967	1846	1881	1846
query48	405	327	320	320
query49	752	410	394	394
query50	804	389	422	389
query51	7262	7153	7028	7028
query52	99	87	89	87
query53	251	181	183	181
query54	516	410	428	410
query55	77	79	77	77
query56	244	261	229	229
query57	1318	1199	1155	1155
query58	214	205	193	193
query59	3138	2974	3006	2974
query60	276	245	239	239
query61	104	97	102	97
query62	775	666	682	666
query63	208	186	187	186
query64	1352	654	643	643
query65	3274	3188	3206	3188
query66	713	303	317	303
query67	15942	15826	15619	15619
query68	3979	571	580	571
query69	410	251	248	248
query70	1217	1104	1126	1104
query71	357	247	255	247
query72	6534	4046	4181	4046
query73	756	357	352	352
query74	9958	8944	9050	8944
query75	3367	2652	2682	2652
query76	1794	1138	1108	1108
query77	502	279	271	271
query78	10287	9393	9431	9393
query79	1814	584	601	584
query80	1373	417	426	417
query81	537	234	243	234
query82	1302	118	118	118
query83	249	142	134	134
query84	279	67	70	67
query85	982	300	289	289
query86	427	304	300	300
query87	4848	4650	4755	4650
query88	3894	2232	2167	2167
query89	411	299	288	288
query90	1941	188	181	181
query91	136	100	110	100
query92	62	49	50	49
query93	2034	553	559	553
query94	841	286	293	286
query95	351	253	244	244
query96	642	278	285	278
query97	2934	2727	2685	2685
query98	212	202	199	199
query99	1627	1316	1323	1316
Total cold run time: 319572 ms
Total hot run time: 195949 ms

doris-robot · 2024-11-04T06:57:17Z

ClickBench: Total hot run time: 33.16 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit b6e22ec5cb82a9ae27d793046c7f474b01e2f833, data reload: false
query1	0.03	0.03	0.03
query2	0.06	0.03	0.03
query3	0.23	0.06	0.06
query4	1.64	0.10	0.10
query5	0.41	0.41	0.40
query6	1.18	0.65	0.65
query7	0.01	0.01	0.03
query8	0.04	0.03	0.03
query9	0.56	0.50	0.49
query10	0.54	0.53	0.56
query11	0.14	0.10	0.10
query12	0.13	0.11	0.11
query13	0.60	0.60	0.59
query14	2.83	2.78	2.75
query15	0.89	0.82	0.82
query16	0.39	0.39	0.37
query17	1.04	1.05	1.04
query18	0.19	0.19	0.20
query19	1.96	1.82	1.92
query20	0.01	0.01	0.01
query21	15.36	0.57	0.59
query22	3.21	2.43	1.67
query23	17.15	0.85	0.82
query24	2.87	2.09	1.87
query25	0.28	0.16	0.12
query26	0.55	0.14	0.13
query27	0.04	0.04	0.03
query28	9.57	1.10	1.06
query29	12.53	3.20	3.21
query30	0.25	0.06	0.06
query31	2.86	0.39	0.39
query32	3.26	0.46	0.45
query33	2.97	3.03	3.08
query34	17.20	4.47	4.44
query35	4.45	4.45	4.43
query36	0.65	0.47	0.47
query37	0.09	0.05	0.06
query38	0.05	0.03	0.04
query39	0.04	0.02	0.02
query40	0.16	0.12	0.12
query41	0.09	0.02	0.02
query42	0.03	0.02	0.02
query43	0.04	0.03	0.03
Total cold run time: 106.58 s
Total hot run time: 33.16 s

csun5285 · 2024-11-04T07:37:03Z

run buildall

csun5285 · 2024-11-04T08:36:49Z

run buildall

doris-robot · 2024-11-04T08:55:21Z

TPC-H: Total hot run time: 41395 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 1bd320d48634b6f55901642a0de22bb60e78fd3a, data reload: false
------ Round 1 ----------------------------------
q1	17573	7483	7359	7359
q2	2053	152	150	150
q3	10594	1119	1147	1119
q4	10233	838	842	838
q5	7745	3056	3028	3028
q6	236	147	142	142
q7	1011	599	599	599
q8	9344	2028	2029	2028
q9	6633	6441	6433	6433
q10	7046	2411	2411	2411
q11	453	256	247	247
q12	407	216	211	211
q13	17756	3031	2978	2978
q14	253	227	213	213
q15	581	537	513	513
q16	674	587	599	587
q17	977	481	462	462
q18	7424	6750	6844	6750
q19	1332	972	995	972
q20	458	175	178	175
q21	3979	3197	3219	3197
q22	1117	1005	983	983
Total cold run time: 107879 ms
Total hot run time: 41395 ms
----- Round 2, with runtime_filter_mode=off -----
q1	7363	7303	7334	7303
q2	323	233	234	233
q3	2908	2765	2804	2765
q4	1930	1726	1689	1689
q5	5408	5471	5504	5471
q6	218	137	137	137
q7	2135	1735	1689	1689
q8	3262	3393	3423	3393
q9	8630	8638	8595	8595
q10	3500	3482	3423	3423
q11	585	504	496	496
q12	764	575	573	573
q13	9717	2996	2990	2990
q14	291	270	273	270
q15	560	511	515	511
q16	660	642	623	623
q17	1823	1584	1600	1584
q18	7785	7530	7351	7351
q19	1670	1607	1570	1570
q20	2058	1846	1870	1846
q21	5396	5344	5078	5078
q22	1101	1024	1001	1001
Total cold run time: 68087 ms
Total hot run time: 58591 ms

doris-robot · 2024-11-04T09:07:26Z

TPC-DS: Total hot run time: 191115 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 1bd320d48634b6f55901642a0de22bb60e78fd3a, data reload: false
query1	985	364	373	364
query2	6523	2068	2030	2030
query3	6786	227	214	214
query4	33884	23537	23532	23532
query5	4309	445	421	421
query6	259	164	160	160
query7	4603	287	287	287
query8	276	218	227	218
query9	9292	2617	2596	2596
query10	468	252	245	245
query11	18387	15132	15252	15132
query12	151	107	101	101
query13	1674	410	403	403
query14	10413	7212	6869	6869
query15	284	172	186	172
query16	8153	470	470	470
query17	1726	570	557	557
query18	2146	302	300	300
query19	358	147	148	147
query20	115	106	109	106
query21	206	99	101	99
query22	4653	4520	4519	4519
query23	34934	34002	34162	34002
query24	11073	2722	2748	2722
query25	660	410	403	403
query26	1405	157	157	157
query27	2815	279	282	279
query28	8038	2435	2415	2415
query29	909	432	444	432
query30	324	158	164	158
query31	1027	808	820	808
query32	95	55	58	55
query33	779	282	263	263
query34	981	506	511	506
query35	883	730	730	730
query36	1093	960	962	960
query37	137	72	75	72
query38	4333	4182	4260	4182
query39	1514	1452	1464	1452
query40	288	105	101	101
query41	49	45	49	45
query42	110	99	94	94
query43	542	487	486	486
query44	1267	796	794	794
query45	183	164	168	164
query46	1143	698	680	680
query47	1985	1859	1835	1835
query48	421	335	338	335
query49	1188	407	398	398
query50	811	394	391	391
query51	7428	7151	6980	6980
query52	104	88	86	86
query53	251	180	178	178
query54	1204	407	413	407
query55	77	78	77	77
query56	258	248	247	247
query57	1312	1180	1182	1180
query58	245	219	216	216
query59	3328	2922	3092	2922
query60	281	249	249	249
query61	105	103	102	102
query62	867	681	688	681
query63	203	183	180	180
query64	5290	632	610	610
query65	3331	3197	3220	3197
query66	1456	306	306	306
query67	16223	15741	15890	15741
query68	4841	552	543	543
query69	446	254	252	252
query70	1173	1115	1102	1102
query71	426	242	237	237
query72	6641	4101	4021	4021
query73	761	357	371	357
query74	10571	9003	9007	9003
query75	3441	2671	2693	2671
query76	2949	1028	1139	1028
query77	433	261	259	259
query78	10370	9474	9383	9383
query79	1316	579	596	579
query80	974	432	413	413
query81	542	236	234	234
query82	1143	118	117	117
query83	220	140	136	136
query84	234	72	71	71
query85	1258	323	280	280
query86	367	292	316	292
query87	4901	4628	4676	4628
query88	3507	2184	2147	2147
query89	399	290	292	290
query90	2025	186	182	182
query91	134	102	102	102
query92	56	49	47	47
query93	1070	546	531	531
query94	994	288	293	288
query95	343	241	243	241
query96	602	271	278	271
query97	2889	2686	2704	2686
query98	220	200	202	200
query99	1526	1333	1304	1304
Total cold run time: 303373 ms
Total hot run time: 191115 ms

doris-robot · 2024-11-04T09:12:54Z

ClickBench: Total hot run time: 33.07 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 1bd320d48634b6f55901642a0de22bb60e78fd3a, data reload: false
query1	0.04	0.04	0.04
query2	0.07	0.03	0.03
query3	0.23	0.06	0.07
query4	1.63	0.11	0.11
query5	0.43	0.41	0.41
query6	1.16	0.66	0.65
query7	0.02	0.02	0.02
query8	0.06	0.03	0.03
query9	0.56	0.49	0.51
query10	0.54	0.56	0.56
query11	0.14	0.10	0.11
query12	0.15	0.11	0.11
query13	0.62	0.60	0.59
query14	2.85	2.78	2.81
query15	0.91	0.82	0.81
query16	0.38	0.39	0.38
query17	1.04	1.07	1.02
query18	0.22	0.22	0.21
query19	1.91	1.82	1.95
query20	0.01	0.00	0.01
query21	15.36	0.58	0.59
query22	2.44	1.60	1.73
query23	17.28	0.93	0.86
query24	3.14	1.54	2.22
query25	0.24	0.13	0.14
query26	0.57	0.14	0.13
query27	0.04	0.05	0.04
query28	9.01	1.09	1.08
query29	12.53	3.24	3.23
query30	0.25	0.06	0.06
query31	2.87	0.38	0.38
query32	3.24	0.46	0.46
query33	2.98	3.02	3.05
query34	17.02	4.47	4.46
query35	4.49	4.51	4.50
query36	0.65	0.48	0.48
query37	0.09	0.06	0.06
query38	0.04	0.04	0.03
query39	0.03	0.02	0.02
query40	0.16	0.12	0.13
query41	0.08	0.02	0.02
query42	0.03	0.03	0.02
query43	0.04	0.03	0.03
Total cold run time: 105.55 s
Total hot run time: 33.07 s

doris-robot · 2024-11-04T10:00:38Z

TeamCity be ut coverage result:
Function Coverage: 37.81% (9821/25973)
Line Coverage: 28.97% (81631/281740)
Region Coverage: 28.22% (42076/149112)
Branch Coverage: 24.80% (21350/86074)
Coverage Report: http://coverage.selectdb-in.cc/coverage/1bd320d48634b6f55901642a0de22bb60e78fd3a_1bd320d48634b6f55901642a0de22bb60e78fd3a/report/index.html

airborne12 · 2024-11-05T07:14:28Z

run p1

airborne12 · 2024-11-05T07:14:34Z

run cloud_p1

airborne12

LGTM

github-actions · 2024-11-05T07:15:45Z

PR approved by at least one committer and no changes requested.

github-actions · 2024-11-05T07:15:48Z

PR approved by anyone and no changes requested.

eldenmoon

LGTM

…t schema (apache#43003) ## Proposed changes refactor inverted index interface in tablet schema 1. There's a large amount of similar code ``` for (const auto& index : schema_ptr->indexes()) { if (index.index_type() == IndexType::INVERTED) { .... } } ``` Now it has been changed to ``` for (const auto& index : schema_ptr->inverted_indexes()) { .... } ``` 2. Remove functions with redundant functionality, such as `get_indexes_for_column `, `get_inverted_index_with_index_id` .... 3. Retrieve the index meta for the inverted index. Currently, there are two types: one is `inverted_index(TabletColumn)`, which checks the column type (e.g., if the column type is float, it does not support indexing and will be filtered out); the other is `inverted_index(unique_id, index_suffix)`, which retrieves the index meta for the current column without checking the column type.

…t schema (#43003) (#43280) (#43669) pick from master #43003,#43280

…t schema (apache#43003) ## Proposed changes refactor inverted index interface in tablet schema 1. There's a large amount of similar code ``` for (const auto& index : schema_ptr->indexes()) { if (index.index_type() == IndexType::INVERTED) { .... } } ``` Now it has been changed to ``` for (const auto& index : schema_ptr->inverted_indexes()) { .... } ``` 2. Remove functions with redundant functionality, such as `get_indexes_for_column `, `get_inverted_index_with_index_id` .... 3. Retrieve the index meta for the inverted index. Currently, there are two types: one is `inverted_index(TabletColumn)`, which checks the column type (e.g., if the column type is float, it does not support indexing and will be filtered out); the other is `inverted_index(unique_id, index_suffix)`, which retrieves the index meta for the current column without checking the column type.

github-actionsBot reviewed Oct 31, 2024
View reviewed changes

Comment threadbe/src/olap/tablet_schema.h
Comment threadbe/test/olap/tablet_index_test.cpp

csun5285 force-pushed the refactor_index_schema branch from 18df067 to 79b8c37Compare October 31, 2024 12:29

eldenmoon reviewed Nov 1, 2024
View reviewed changes

Comment threadbe/src/olap/rowset/segment_v2/segment_iterator.cpp

airborne12 reviewed Nov 1, 2024
View reviewed changes

Comment threadbe/src/cloud/cloud_warm_up_manager.cpp

csun5285 added 3 commits November 4, 2024 13:00

[Refactor](inverted index) refactor inverted index interface in table…
bc8f2bd
…t schema

fix rebase
504fef2

fix
b6e22ec

csun5285 force-pushed the refactor_index_schema branch from 70bca58 to b6e22ecCompare November 4, 2024 05:00

xiaokang reviewed Nov 4, 2024
View reviewed changes

fix comment
5150ca7

fix
1bd320d

airborne12 approved these changes Nov 5, 2024
View reviewed changes

github-actionsBot added approved Indicates a PR has been approved by one committer. reviewed labels Nov 5, 2024

eldenmoon approved these changes Nov 5, 2024
View reviewed changes

airborne12 merged commit 8db12f3 into apache:masterNov 5, 2024

airborne12 added the dev/3.0.x label Nov 5, 2024

csun5285 mentioned this pull request Nov 6, 2024
[fix] (UT) fix BE UT #43280
Merged
16 tasks

airborne12 removed the dev/3.0.x label Nov 7, 2024

csun5285 mentioned this pull request Nov 12, 2024
[Refactor](inverted index) refactor inverted index interface in tablet schema (#43003) (#43280) #43669
Merged

airborne12 pushed a commit that referenced this pull request Nov 12, 2024
[Refactor](inverted index) refactor inverted index interface in table…
9c672fb
…t schema (#43003) (#43280) (#43669) pick from master #43003,#43280

airborne12 added the dev/3.0.3-merged label Nov 12, 2024

Uh oh!

Conversation

csun5285 commented Oct 31, 2024• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Proposed changes

Uh oh!

doris-robot commented Oct 31, 2024

Uh oh!

csun5285 commented Oct 31, 2024

Uh oh!

github-actionsBot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

doris-robot commented Oct 31, 2024

Uh oh!

doris-robot commented Oct 31, 2024

Uh oh!

doris-robot commented Oct 31, 2024

Uh oh!

doris-robot commented Oct 31, 2024

Uh oh!

csun5285 commented Oct 31, 2024

Uh oh!

csun5285 commented Oct 31, 2024

Uh oh!

csun5285 commented Oct 31, 2024

Uh oh!

doris-robot commented Oct 31, 2024

Uh oh!

doris-robot commented Nov 1, 2024

Uh oh!

doris-robot commented Nov 1, 2024

Uh oh!

doris-robot commented Nov 1, 2024

Uh oh!

Uh oh!

Uh oh!

csun5285 commented Nov 4, 2024

Uh oh!

doris-robot commented Nov 4, 2024

Uh oh!

doris-robot commented Nov 4, 2024

Uh oh!

doris-robot commented Nov 4, 2024

Uh oh!

doris-robot commented Nov 4, 2024

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

csun5285 commented Nov 4, 2024

Uh oh!

csun5285 commented Nov 4, 2024

Uh oh!

doris-robot commented Nov 4, 2024

Uh oh!

doris-robot commented Nov 4, 2024

Uh oh!

doris-robot commented Nov 4, 2024

Uh oh!

doris-robot commented Nov 4, 2024

Uh oh!

airborne12 commented Nov 5, 2024

Uh oh!

airborne12 commented Nov 5, 2024

Uh oh!

airborne12 left a comment

Choose a reason for hiding this comment

Uh oh!

github-actionsBot commented Nov 5, 2024

Uh oh!

github-actionsBot commented Nov 5, 2024

Uh oh!

eldenmoon left a comment

Choose a reason for hiding this comment

csun5285 commented Oct 31, 2024•
edited
Loading