[fix](inverted index) warm up inverted index by zzzxl1993 · Pull Request #38986 · apache/doris

zzzxl1993 · 2024-08-06T16:37:37Z

Proposed changes

Issue Number: close #xxx

doris-robot · 2024-08-06T16:37:42Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

zzzxl1993 · 2024-08-06T16:37:44Z

run buildall

github-actions · 2024-08-06T16:44:30Z

clang-tidy review says "All clean, LGTM! 👍"

doris-robot · 2024-08-06T17:41:44Z

TPC-H: Total hot run time: 41854 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 9744b0faf178de7a692554771087ba15c6be8e16, data reload: false
------ Round 1 ----------------------------------
q1	17711	4221	4119	4119
q2	2024	204	202	202
q3	10593	1267	1356	1267
q4	10250	844	966	844
q5	7634	3097	3017	3017
q6	222	136	136	136
q7	1062	614	607	607
q8	9454	1916	1965	1916
q9	8547	6667	6625	6625
q10	8800	3872	3859	3859
q11	435	252	251	251
q12	415	224	225	224
q13	17764	2920	2931	2920
q14	272	249	247	247
q15	528	494	497	494
q16	527	412	390	390
q17	975	949	933	933
q18	8053	7452	7221	7221
q19	3364	1198	1221	1198
q20	567	347	332	332
q21	5378	4822	4769	4769
q22	354	284	283	283
Total cold run time: 114929 ms
Total hot run time: 41854 ms
----- Round 2, with runtime_filter_mode=off -----
q1	4158	4093	4034	4034
q2	333	221	218	218
q3	3016	3169	3222	3169
q4	2013	1975	2053	1975
q5	5580	5500	5435	5435
q6	222	131	130	130
q7	2220	1834	1780	1780
q8	3316	3412	3365	3365
q9	8711	8849	8729	8729
q10	4021	3858	4040	3858
q11	579	471	465	465
q12	761	615	578	578
q13	16358	3137	3146	3137
q14	312	271	270	270
q15	540	492	484	484
q16	506	419	436	419
q17	1789	1756	1741	1741
q18	8296	7875	7849	7849
q19	1765	1750	1746	1746
q20	2073	1866	1823	1823
q21	5704	5439	5344	5344
q22	553	458	456	456
Total cold run time: 72826 ms
Total hot run time: 57005 ms

doris-robot · 2024-08-06T17:52:31Z

TPC-DS: Total hot run time: 170506 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 9744b0faf178de7a692554771087ba15c6be8e16, data reload: false
query1	916	383	368	368
query2	6473	1754	1751	1751
query3	6637	213	221	213
query4	19494	18120	17863	17863
query5	3647	510	522	510
query6	267	168	163	163
query7	4590	304	305	304
query8	263	216	194	194
query9	8518	2405	2371	2371
query10	436	289	266	266
query11	10621	10084	10052	10052
query12	128	92	98	92
query13	1631	375	376	375
query14	9224	7472	6903	6903
query15	203	177	170	170
query16	7016	459	462	459
query17	980	615	562	562
query18	1932	297	299	297
query19	198	150	148	148
query20	96	84	91	84
query21	210	104	98	98
query22	4119	4138	3908	3908
query23	34137	33722	33676	33676
query24	10171	3086	3127	3086
query25	683	446	430	430
query26	1581	161	163	161
query27	3141	285	287	285
query28	7366	2049	2026	2026
query29	1127	450	434	434
query30	233	162	161	161
query31	972	776	800	776
query32	100	61	57	57
query33	687	342	329	329
query34	939	488	506	488
query35	867	759	754	754
query36	1054	869	837	837
query37	161	78	77	77
query38	2888	2759	2780	2759
query39	874	823	817	817
query40	250	112	111	111
query41	48	44	44	44
query42	119	103	98	98
query43	469	421	423	421
query44	1175	730	744	730
query45	214	182	185	182
query46	1096	865	785	785
query47	1818	1715	1748	1715
query48	359	287	285	285
query49	889	428	420	420
query50	910	435	429	429
query51	6816	6679	6566	6566
query52	106	90	90	90
query53	254	185	185	185
query54	613	470	450	450
query55	79	74	73	73
query56	306	251	261	251
query57	1147	1070	1046	1046
query58	271	263	274	263
query59	2560	2368	2561	2368
query60	333	285	266	266
query61	97	107	98	98
query62	875	657	656	656
query63	208	189	186	186
query64	5653	1973	1881	1881
query65	3182	3113	3117	3113
query66	1288	336	338	336
query67	15235	15061	14912	14912
query68	4474	573	577	573
query69	694	379	318	318
query70	1041	1051	1063	1051
query71	473	278	273	273
query72	7760	2665	2525	2525
query73	806	332	324	324
query74	5984	5704	5659	5659
query75	3477	2728	2743	2728
query76	3013	1241	1283	1241
query77	584	310	311	310
query78	9497	8968	9009	8968
query79	2565	534	531	531
query80	1276	513	503	503
query81	546	225	227	225
query82	766	134	130	130
query83	236	172	168	168
query84	255	83	80	80
query85	1374	371	306	306
query86	477	278	300	278
query87	3280	3090	3168	3090
query88	3763	2388	2411	2388
query89	388	291	283	283
query90	1822	190	190	190
query91	125	103	102	102
query92	59	52	49	49
query93	3760	621	608	608
query94	789	311	284	284
query95	381	271	262	262
query96	613	289	282	282
query97	3268	3074	3048	3048
query98	219	204	207	204
query99	1680	1261	1283	1261
Total cold run time: 266395 ms
Total hot run time: 170506 ms

doris-robot · 2024-08-06T17:58:01Z

ClickBench: Total hot run time: 30.12 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 9744b0faf178de7a692554771087ba15c6be8e16, data reload: false
query1	0.05	0.05	0.04
query2	0.08	0.04	0.04
query3	0.22	0.04	0.05
query4	1.69	0.09	0.07
query5	0.49	0.48	0.49
query6	1.13	0.71	0.72
query7	0.02	0.02	0.01
query8	0.05	0.04	0.05
query9	0.56	0.51	0.51
query10	0.57	0.57	0.57
query11	0.16	0.12	0.12
query12	0.14	0.12	0.12
query13	0.61	0.60	0.60
query14	0.77	0.81	0.80
query15	0.91	0.88	0.87
query16	0.36	0.36	0.36
query17	0.99	1.05	1.05
query18	0.22	0.21	0.22
query19	1.89	1.73	1.77
query20	0.01	0.01	0.01
query21	15.40	0.76	0.65
query22	4.20	7.86	1.28
query23	17.80	1.31	1.30
query24	2.21	0.22	0.22
query25	0.18	0.09	0.08
query26	0.33	0.22	0.22
query27	0.47	0.23	0.23
query28	13.17	1.00	0.98
query29	12.56	3.26	3.27
query30	0.26	0.06	0.06
query31	2.87	0.41	0.40
query32	3.25	0.50	0.49
query33	2.98	2.96	2.97
query34	15.45	4.26	4.25
query35	4.31	4.30	4.29
query36	0.66	0.49	0.48
query37	0.19	0.17	0.16
query38	0.15	0.16	0.16
query39	0.04	0.04	0.03
query40	0.15	0.12	0.13
query41	0.09	0.06	0.05
query42	0.05	0.05	0.04
query43	0.04	0.03	0.04
Total cold run time: 107.73 s
Total hot run time: 30.12 s

csun5285 · 2024-08-07T02:36:22Z

+ };
+ _engine.file_cache_block_downloader().submit_download_task(std::move(meta));
+ };
+ auto idx_version = rs->tablet_schema()->get_inverted_index_storage_format();


if (rs->tablet_schema()->has_inverted_index())

csun5285 · 2024-08-07T02:41:24Z

 });
+
+ // idx
+ auto download_idx_file = [&](const io::Path& idx_path) {


The CloudTablet::add_rowsets also requires this.

gavinchou · 2024-08-07T07:52:01Z

 private:
 void download_file_cache_block(const DownloadTask::FileCacheBlockMetaVec&);
- void download_segment_file(const DownloadFileMeta&);
+ void download_file(const DownloadFileMeta&);


why rename?

github-actions · 2024-08-07T10:55:34Z

clang-tidy review says "All clean, LGTM! 👍"

zzzxl1993 · 2024-08-08T04:19:29Z

run buildall

github-actions · 2024-08-08T04:25:11Z

clang-tidy review says "All clean, LGTM! 👍"

csun5285

LGTM

github-actions · 2024-08-08T04:27:09Z

PR approved by anyone and no changes requested.

doris-robot · 2024-08-08T05:05:23Z

TPC-H: Total hot run time: 39965 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 7bc6672e56277efc0443b749c72f0ac7e83abb44, data reload: false
------ Round 1 ----------------------------------
q1	17903	4771	4314	4314
q2	2021	174	172	172
q3	10528	1228	1075	1075
q4	10153	852	771	771
q5	7590	2615	2569	2569
q6	221	135	137	135
q7	991	588	588	588
q8	9219	1981	1941	1941
q9	8776	6664	6628	6628
q10	7056	2199	2241	2199
q11	452	251	248	248
q12	400	214	215	214
q13	17903	2978	2994	2978
q14	288	230	242	230
q15	530	478	496	478
q16	516	401	394	394
q17	1006	708	755	708
q18	8259	7657	7402	7402
q19	4918	1040	1132	1040
q20	738	324	327	324
q21	5466	4577	4537	4537
q22	1129	1034	1020	1020
Total cold run time: 116063 ms
Total hot run time: 39965 ms
----- Round 2, with runtime_filter_mode=off -----
q1	4484	4399	4326	4326
q2	378	258	266	258
q3	2897	2655	2631	2631
q4	1887	1643	1605	1605
q5	5257	5264	5284	5264
q6	220	131	130	130
q7	2065	1624	1662	1624
q8	3199	3391	3402	3391
q9	8455	8463	8445	8445
q10	3413	3140	3142	3140
q11	593	496	496	496
q12	758	633	627	627
q13	17519	2983	3004	2983
q14	295	272	276	272
q15	528	481	474	474
q16	474	408	414	408
q17	1822	1489	1473	1473
q18	7734	7575	7460	7460
q19	1699	1438	1560	1438
q20	2034	1800	1838	1800
q21	5347	5009	5172	5009
q22	1105	1029	1022	1022
Total cold run time: 72163 ms
Total hot run time: 54276 ms

doris-robot · 2024-08-08T12:17:05Z

TPC-H: Total hot run time: 39746 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 7bc6672e56277efc0443b749c72f0ac7e83abb44, data reload: false
------ Round 1 ----------------------------------
q1	17640	4466	4350	4350
q2	2031	179	180	179
q3	10487	1197	1161	1161
q4	10144	701	674	674
q5	7496	2531	2539	2531
q6	232	145	146	145
q7	961	591	602	591
q8	9249	1927	1942	1927
q9	8821	6607	6558	6558
q10	7069	2195	2187	2187
q11	466	245	253	245
q12	497	230	223	223
q13	18741	2987	3005	2987
q14	271	231	235	231
q15	566	489	467	467
q16	512	385	384	384
q17	984	726	724	724
q18	8176	7517	7351	7351
q19	1376	1044	1065	1044
q20	689	316	326	316
q21	5677	4535	4484	4484
q22	1109	987	1010	987
Total cold run time: 113194 ms
Total hot run time: 39746 ms
----- Round 2, with runtime_filter_mode=off -----
q1	4346	4245	4262	4245
q2	369	273	265	265
q3	2880	2654	2597	2597
q4	1894	1662	1643	1643
q5	5329	5283	5267	5267
q6	225	130	132	130
q7	2074	1678	1666	1666
q8	3175	3383	3347	3347
q9	8445	8343	8380	8343
q10	3420	3156	3172	3156
q11	595	518	509	509
q12	791	620	602	602
q13	17624	2969	2975	2969
q14	300	272	269	269
q15	523	488	484	484
q16	473	420	408	408
q17	1782	1479	1508	1479
q18	7598	7679	7268	7268
q19	1710	1432	1539	1432
q20	1982	1784	1801	1784
q21	5208	5112	5185	5112
q22	1094	1020	978	978
Total cold run time: 71837 ms
Total hot run time: 53953 ms

doris-robot · 2024-08-08T12:29:47Z

TPC-DS: Total hot run time: 201626 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 7bc6672e56277efc0443b749c72f0ac7e83abb44, data reload: false
query1	954	413	381	381
query2	6447	1876	1796	1796
query3	6638	211	227	211
query4	34359	23188	23218	23188
query5	4201	496	487	487
query6	292	184	185	184
query7	4580	298	303	298
query8	242	213	205	205
query9	8599	2394	2375	2375
query10	967	889	919	889
query11	16709	15051	14977	14977
query12	149	97	97	97
query13	1635	402	377	377
query14	10551	7215	8466	7215
query15	359	355	371	355
query16	7655	434	466	434
query17	1726	571	547	547
query18	1615	393	372	372
query19	253	223	233	223
query20	116	107	107	107
query21	205	106	106	106
query22	4483	4052	3928	3928
query23	33866	33450	33085	33085
query24	11865	2898	2829	2829
query25	621	364	376	364
query26	1574	152	153	152
query27	2665	279	283	279
query28	6980	2011	2001	2001
query29	919	406	399	399
query30	302	145	146	145
query31	966	730	729	729
query32	91	53	59	53
query33	724	289	271	271
query34	940	464	486	464
query35	942	858	848	848
query36	1074	927	916	916
query37	262	84	83	83
query38	4276	4143	4239	4143
query39	1444	1372	1388	1372
query40	268	115	115	115
query41	51	44	44	44
query42	116	95	97	95
query43	490	465	461	461
query44	1174	741	752	741
query45	415	379	362	362
query46	1111	761	769	761
query47	1835	1800	1802	1800
query48	375	296	303	296
query49	1165	426	430	426
query50	809	406	407	406
query51	6788	6677	6647	6647
query52	106	84	96	84
query53	259	188	182	182
query54	982	452	447	447
query55	76	73	74	73
query56	270	242	247	242
query57	1167	1074	1054	1054
query58	232	233	231	231
query59	2761	2686	2693	2686
query60	287	263	269	263
query61	96	95	95	95
query62	823	652	664	652
query63	220	184	180	180
query64	10613	2383	1915	1915
query65	3156	3142	3160	3142
query66	1233	350	331	331
query67	15073	14894	14860	14860
query68	4701	556	550	550
query69	413	375	423	375
query70	1214	1128	1142	1128
query71	466	314	265	265
query72	19537	16753	16265	16265
query73	769	335	337	335
query74	9072	8790	8718	8718
query75	3873	2679	2641	2641
query76	3288	979	963	963
query77	647	309	310	309
query78	10384	8985	9029	8985
query79	2067	507	510	507
query80	1120	494	485	485
query81	598	226	232	226
query82	757	137	136	136
query83	312	150	144	144
query84	260	81	80	80
query85	1985	321	356	321
query86	451	269	299	269
query87	4651	4591	4484	4484
query88	4433	2492	2488	2488
query89	400	282	286	282
query90	1924	206	198	198
query91	139	113	123	113
query92	67	49	49	49
query93	2406	528	544	528
query94	977	302	279	279
query95	355	260	263	260
query96	615	275	281	275
query97	3256	3070	3060	3060
query98	235	198	193	193
query99	1479	1264	1239	1239
Total cold run time: 317048 ms
Total hot run time: 201626 ms

doris-robot · 2024-08-08T12:35:18Z

ClickBench: Total hot run time: 31.18 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 7bc6672e56277efc0443b749c72f0ac7e83abb44, data reload: false
query1	0.04	0.04	0.04
query2	0.08	0.04	0.04
query3	0.22	0.06	0.06
query4	1.66	0.08	0.08
query5	0.51	0.48	0.48
query6	1.12	0.73	0.74
query7	0.02	0.01	0.01
query8	0.05	0.05	0.05
query9	0.54	0.48	0.48
query10	0.54	0.54	0.54
query11	0.16	0.11	0.11
query12	0.15	0.12	0.12
query13	0.60	0.59	0.59
query14	0.78	0.78	0.80
query15	0.86	0.82	0.84
query16	0.37	0.37	0.37
query17	1.01	1.00	0.98
query18	0.23	0.22	0.22
query19	1.84	1.71	1.73
query20	0.01	0.01	0.01
query21	15.43	0.75	0.65
query22	3.97	6.48	2.30
query23	18.32	1.47	1.37
query24	2.09	0.22	0.21
query25	0.16	0.08	0.09
query26	0.30	0.21	0.20
query27	0.46	0.22	0.22
query28	13.31	1.03	1.00
query29	12.59	3.28	3.28
query30	0.23	0.05	0.05
query31	2.91	0.40	0.39
query32	3.27	0.50	0.48
query33	2.88	2.97	2.91
query34	17.07	4.34	4.36
query35	4.42	4.49	4.43
query36	0.65	0.48	0.46
query37	0.19	0.15	0.15
query38	0.15	0.14	0.14
query39	0.04	0.04	0.04
query40	0.15	0.13	0.13
query41	0.09	0.05	0.05
query42	0.06	0.05	0.04
query43	0.04	0.04	0.04
Total cold run time: 109.57 s
Total hot run time: 31.18 s

qidaye

LGTM

github-actions · 2024-08-09T04:10:50Z

PR approved by at least one committer and no changes requested.

github-actionsBot added the doing label Aug 6, 2024

csun5285 requested changes Aug 7, 2024
View reviewed changes

qidaye requested a review from platoneko August 7, 2024 02:42

gavinchou reviewed Aug 7, 2024
View reviewed changes

zzzxl1993 force-pushed the 202408071235 branch from 9744b0f to 508cc32Compare August 7, 2024 10:48

[fix](inverted index) warm up inverted index
7bc6672

zzzxl1993 force-pushed the 202408071235 branch from 508cc32 to 7bc6672Compare August 8, 2024 04:18

csun5285 approved these changes Aug 8, 2024
View reviewed changes

github-actionsBot added the reviewed label Aug 8, 2024

qidaye approved these changes Aug 9, 2024
View reviewed changes

github-actionsBot added the approved Indicates a PR has been approved by one committer. label Aug 9, 2024

qidaye merged commit 5028bad into apache:masterAug 9, 2024

dataroaring pushed a commit that referenced this pull request Aug 11, 2024
[fix](inverted index) warm up inverted index (#38986)
90e7d86

dataroaring added the dev/3.0.2-merged label Aug 12, 2024

wyxxxcat pushed a commit to wyxxxcat/doris that referenced this pull request Aug 14, 2024
[fix](inverted index) warm up inverted index (apache#38986)
5f1b47c

xiaokang removed the doing label Aug 14, 2024

dataroaring pushed a commit that referenced this pull request Aug 16, 2024
[fix](inverted index) warm up inverted index (#38986)
22a0702

gavinchou mentioned this pull request Oct 13, 2024
Release Note 3.0.2 #41558
Closed

HappenLee pushed a commit to HappenLee/incubator-doris that referenced this pull request Apr 24, 2026
[fix](inverted index) warm up inverted index (apache#38986)
6e6e630

Uh oh!

Conversation

zzzxl1993 commented Aug 6, 2024

Proposed changes

Uh oh!

doris-robot commented Aug 6, 2024

Uh oh!

zzzxl1993 commented Aug 6, 2024

Uh oh!

github-actionsBot commented Aug 6, 2024

Uh oh!

doris-robot commented Aug 6, 2024

Uh oh!

doris-robot commented Aug 6, 2024

Uh oh!

doris-robot commented Aug 6, 2024

Uh oh!

csun5285Aug 7, 2024

Choose a reason for hiding this comment

Uh oh!

csun5285Aug 7, 2024

Choose a reason for hiding this comment

Uh oh!

gavinchouAug 7, 2024

Choose a reason for hiding this comment

Uh oh!

github-actionsBot commented Aug 7, 2024

Uh oh!

zzzxl1993 commented Aug 8, 2024

Uh oh!

github-actionsBot commented Aug 8, 2024

Uh oh!

csun5285 left a comment

Choose a reason for hiding this comment

Uh oh!

github-actionsBot commented Aug 8, 2024

Uh oh!

doris-robot commented Aug 8, 2024

Uh oh!

doris-robot commented Aug 8, 2024

Uh oh!

doris-robot commented Aug 8, 2024

Uh oh!

doris-robot commented Aug 8, 2024

Uh oh!

qidaye left a comment

Choose a reason for hiding this comment

Uh oh!

github-actionsBot commented Aug 9, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants