[fix](inverted index) Writing to the inverted index also writes to the file cache. by zzzxl1993 · Pull Request #39076 · apache/doris

zzzxl1993 · 2024-08-08T03:17:33Z

Proposed changes

When write_file_cache is true, writing to the inverted index also writes to the file cache.

doris-robot · 2024-08-08T03:17:38Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

github-actions · 2024-08-08T03:24:17Z

clang-tidy review says "All clean, LGTM! 👍"

zzzxl1993 · 2024-08-08T13:04:14Z

run buildall

github-actions · 2024-08-08T13:05:33Z

clang-tidy review says "All clean, LGTM! 👍"

github-actions · 2024-08-08T13:07:53Z

clang-tidy review says "All clean, LGTM! 👍"

github-actions · 2024-08-08T13:10:54Z

clang-tidy review says "All clean, LGTM! 👍"

doris-robot · 2024-08-08T13:28:49Z

TPC-H: Total hot run time: 39692 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 7956e1678a5be2bde57b45c40d1954736274c173, data reload: false
------ Round 1 ----------------------------------
q1	17606	4326	4356	4326
q2	2026	182	184	182
q3	10463	1253	1086	1086
q4	10128	708	802	708
q5	7551	2551	2541	2541
q6	225	139	140	139
q7	965	594	615	594
q8	9409	1911	1917	1911
q9	10212	6580	6598	6580
q10	7063	2153	2243	2153
q11	447	251	256	251
q12	500	223	232	223
q13	19011	2964	2959	2959
q14	284	232	237	232
q15	541	483	499	483
q16	505	397	390	390
q17	984	732	668	668
q18	8061	7482	7480	7480
q19	1385	962	1006	962
q20	649	335	339	335
q21	5307	4497	4628	4497
q22	1104	1024	992	992
Total cold run time: 114426 ms
Total hot run time: 39692 ms
----- Round 2, with runtime_filter_mode=off -----
q1	4311	4292	4274	4274
q2	367	285	276	276
q3	2844	2658	2598	2598
q4	1857	1558	1555	1555
q5	5207	5260	5295	5260
q6	216	137	132	132
q7	2023	1670	1675	1670
q8	3151	3343	3330	3330
q9	8423	8328	8368	8328
q10	3386	3121	3186	3121
q11	590	482	483	482
q12	761	606	629	606
q13	16772	2959	2978	2959
q14	299	260	274	260
q15	518	473	480	473
q16	470	407	412	407
q17	1778	1493	1461	1461
q18	7603	7592	7296	7296
q19	1634	1571	1526	1526
q20	1979	1788	1813	1788
q21	5324	5078	5088	5078
q22	1080	1011	998	998
Total cold run time: 70593 ms
Total hot run time: 53878 ms

github-actions · 2024-08-09T08:09:40Z

clang-tidy review says "All clean, LGTM! 👍"

github-actions · 2024-08-09T09:05:54Z

clang-tidy review says "All clean, LGTM! 👍"

zzzxl1993 · 2024-08-09T15:46:03Z

run buildall

github-actions · 2024-08-12T08:45:18Z

clang-tidy review says "All clean, LGTM! 👍"

zzzxl1993 · 2024-08-13T03:49:16Z

run buildall

github-actions · 2024-08-13T03:56:22Z

clang-tidy review says "All clean, LGTM! 👍"

doris-robot · 2024-08-13T04:27:24Z

TPC-H: Total hot run time: 40079 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 84e550c41bb32a8fdfc9aec7d43583c10fc237a7, data reload: false
------ Round 1 ----------------------------------
q1	17628	4425	4294	4294
q2	2012	184	179	179
q3	11818	1012	1121	1012
q4	10511	739	759	739
q5	7768	2853	2857	2853
q6	229	143	140	140
q7	965	597	605	597
q8	9550	2073	2061	2061
q9	9990	6590	6623	6590
q10	7083	2218	2210	2210
q11	457	248	249	248
q12	411	241	229	229
q13	17752	3005	3026	3005
q14	283	232	228	228
q15	525	472	489	472
q16	503	403	387	387
q17	990	667	648	648
q18	8114	7548	7396	7396
q19	7552	1055	977	977
q20	668	345	337	337
q21	5359	4653	4471	4471
q22	1103	1006	1022	1006
Total cold run time: 121271 ms
Total hot run time: 40079 ms
----- Round 2, with runtime_filter_mode=off -----
q1	4522	4307	4285	4285
q2	379	279	278	278
q3	3013	2833	2804	2804
q4	2006	1763	1701	1701
q5	5611	5584	5685	5584
q6	232	141	144	141
q7	2194	1777	1836	1777
q8	3339	3472	3428	3428
q9	8901	8940	8862	8862
q10	3439	3325	3294	3294
q11	612	525	529	525
q12	805	616	631	616
q13	16925	3218	3187	3187
q14	344	301	292	292
q15	562	499	482	482
q16	515	440	457	440
q17	1793	1532	1536	1532
q18	8261	8007	8068	8007
q19	1847	1696	1708	1696
q20	3067	1888	1878	1878
q21	8889	5210	5423	5210
q22	1132	1034	1054	1034
Total cold run time: 78388 ms
Total hot run time: 57053 ms

doris-robot · 2024-08-13T04:39:50Z

TPC-DS: Total hot run time: 190847 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 84e550c41bb32a8fdfc9aec7d43583c10fc237a7, data reload: false
query1	1260	882	863	863
query2	6314	1888	1862	1862
query3	10614	4009	3528	3528
query4	59706	24802	23288	23288
query5	5487	521	483	483
query6	492	160	166	160
query7	6421	294	308	294
query8	312	205	213	205
query9	8794	2458	2452	2452
query10	503	273	266	266
query11	16604	15056	15066	15056
query12	161	101	97	97
query13	1581	394	385	385
query14	12640	8031	8129	8031
query15	247	172	182	172
query16	7278	484	483	483
query17	1162	626	606	606
query18	1969	323	304	304
query19	302	166	157	157
query20	127	115	114	114
query21	212	108	104	104
query22	4426	4258	4382	4258
query23	34294	33383	33171	33171
query24	5573	2893	2879	2879
query25	556	408	434	408
query26	695	164	163	163
query27	1793	278	278	278
query28	3874	2057	2048	2048
query29	704	436	440	436
query30	240	157	152	152
query31	909	758	764	758
query32	81	55	57	55
query33	493	304	301	301
query34	851	475	488	475
query35	804	715	753	715
query36	1054	940	925	925
query37	139	81	82	81
query38	3877	3936	3787	3787
query39	1425	1400	1407	1400
query40	205	126	119	119
query41	49	46	47	46
query42	114	99	103	99
query43	497	461	486	461
query44	1070	739	748	739
query45	195	166	164	164
query46	1100	757	742	742
query47	1896	1732	1736	1732
query48	366	303	293	293
query49	775	436	448	436
query50	812	418	422	418
query51	6833	6797	6688	6688
query52	107	92	91	91
query53	256	188	185	185
query54	568	466	449	449
query55	76	75	76	75
query56	265	254	271	254
query57	1154	1039	1039	1039
query58	223	224	232	224
query59	2882	2718	2761	2718
query60	299	273	263	263
query61	112	96	111	96
query62	736	639	646	639
query63	228	187	185	185
query64	3341	1782	1758	1758
query65	3189	3128	3171	3128
query66	682	335	329	329
query67	15437	14979	14728	14728
query68	4920	546	558	546
query69	578	297	272	272
query70	1183	1137	1120	1120
query71	511	280	280	280
query72	7092	2283	2040	2040
query73	800	330	340	330
query74	9105	8778	8767	8767
query75	4795	2693	2717	2693
query76	3344	1063	983	983
query77	769	316	320	316
query78	9773	9138	8949	8949
query79	3038	532	524	524
query80	2537	493	495	493
query81	565	224	224	224
query82	775	137	134	134
query83	303	142	143	142
query84	279	84	77	77
query85	1491	318	276	276
query86	461	306	304	304
query87	4399	4210	4154	4154
query88	4381	2525	2414	2414
query89	402	284	285	284
query90	1992	202	195	195
query91	125	114	100	100
query92	67	50	53	50
query93	4227	542	529	529
query94	990	301	289	289
query95	353	269	273	269
query96	626	280	277	277
query97	3237	3052	3052	3052
query98	231	203	204	203
query99	1678	1272	1263	1263
Total cold run time: 322468 ms
Total hot run time: 190847 ms

doris-robot · 2024-08-13T04:45:22Z

ClickBench: Total hot run time: 30.51 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 84e550c41bb32a8fdfc9aec7d43583c10fc237a7, data reload: false
query1	0.04	0.04	0.04
query2	0.08	0.04	0.04
query3	0.22	0.04	0.05
query4	1.69	0.07	0.08
query5	0.50	0.52	0.48
query6	1.13	0.71	0.73
query7	0.02	0.01	0.01
query8	0.05	0.04	0.04
query9	0.54	0.47	0.48
query10	0.54	0.55	0.54
query11	0.15	0.12	0.11
query12	0.16	0.12	0.12
query13	0.59	0.59	0.58
query14	0.77	0.77	0.79
query15	0.87	0.81	0.81
query16	0.38	0.36	0.35
query17	0.98	0.97	1.01
query18	0.22	0.21	0.22
query19	1.89	1.75	1.76
query20	0.02	0.01	0.01
query21	15.39	0.76	0.67
query22	4.34	7.11	1.76
query23	18.25	1.34	1.29
query24	2.12	0.22	0.22
query25	0.14	0.08	0.10
query26	0.31	0.22	0.22
query27	0.45	0.22	0.22
query28	13.37	1.03	1.00
query29	12.62	3.33	3.29
query30	0.24	0.05	0.05
query31	2.90	0.39	0.39
query32	3.27	0.48	0.47
query33	2.94	3.03	2.96
query34	17.10	4.34	4.39
query35	4.38	4.39	4.40
query36	0.66	0.47	0.46
query37	0.19	0.16	0.15
query38	0.16	0.15	0.14
query39	0.05	0.04	0.04
query40	0.15	0.12	0.13
query41	0.09	0.05	0.05
query42	0.06	0.04	0.05
query43	0.05	0.05	0.04
Total cold run time: 110.07 s
Total hot run time: 30.51 s

airborne12

LGTM

github-actions · 2024-08-13T08:15:55Z

PR approved by at least one committer and no changes requested.

github-actions · 2024-08-13T08:15:57Z

PR approved by anyone and no changes requested.

qidaye

LGTM

…e file cache. (apache#39076) 1. When write_file_cache is true, writing to the inverted index also writes to the file cache.

…e file cache. (#39076) 1. When write_file_cache is true, writing to the inverted index also writes to the file cache.

…40254) introduced by: #39076

…pache#40254) introduced by: apache#39076

…e file cache. (apache#39076) 1. When write_file_cache is true, writing to the inverted index also writes to the file cache.

…pache#40254) introduced by: apache#39076

github-actionsBot added the doing label Aug 8, 2024

zzzxl1993 force-pushed the 202408081039 branch from 5328edf to 7956e16Compare August 8, 2024 13:03

zzzxl1993 force-pushed the 202408081039 branch from 7956e16 to a85c14cCompare August 9, 2024 08:02

zzzxl1993 force-pushed the 202408081039 branch from a85c14c to 6827ad7Compare August 9, 2024 08:58

zzzxl1993 force-pushed the 202408081039 branch from 6827ad7 to a9a7fffCompare August 12, 2024 08:37

zzzxl1993 changed the title ~~[fix](inverted index) The inverted index is written into the file cache~~[fix](inverted index) while writing the inverted index, it is simultaneously written to the file cache.Aug 13, 2024

zzzxl1993 changed the title ~~[fix](inverted index) while writing the inverted index, it is simultaneously written to the file cache.~~[fix](inverted index) Writing to the inverted index also writes to the file cache.Aug 13, 2024

[fix](inverted index) The inverted index is written into the file cache
84e550c

zzzxl1993 force-pushed the 202408081039 branch from a9a7fff to 84e550cCompare August 13, 2024 03:49

airborne12 approved these changes Aug 13, 2024
View reviewed changes

airborne12 added the dev/3.0.x label Aug 13, 2024

github-actionsBot added the approved Indicates a PR has been approved by one committer. label Aug 13, 2024

github-actionsBot added the reviewed label Aug 13, 2024

qidaye approved these changes Aug 13, 2024
View reviewed changes

qidaye merged commit b0519d2 into apache:masterAug 13, 2024

xiaokang removed the doing label Aug 14, 2024

dataroaring pushed a commit that referenced this pull request Aug 17, 2024
[fix](inverted index) Writing to the inverted index also writes to th…
f30529a
…e file cache. (#39076) 1. When write_file_cache is true, writing to the inverted index also writes to the file cache.

dataroaring added dev/3.0.2-merged and removed dev/3.0.x labels Aug 17, 2024

qidaye mentioned this pull request Sep 2, 2024
[fix](index compaction)Fix core when having multiple dest segments #40254
Merged

qidaye added a commit that referenced this pull request Sep 3, 2024
[fix](index compaction)Fix core when having multiple dest segments (#…
ee3be4f
…40254) introduced by: #39076

qidaye added a commit to qidaye/incubator-doris that referenced this pull request Sep 3, 2024
[fix](index compaction)Fix core when having multiple dest segments (a…
58484a5
…pache#40254) introduced by: apache#39076

gavinchou mentioned this pull request Oct 13, 2024
Release Note 3.0.2 #41558
Closed

HappenLee pushed a commit to HappenLee/incubator-doris that referenced this pull request Apr 24, 2026
[fix](index compaction)Fix core when having multiple dest segments (a…
e234c7d
…pache#40254) introduced by: apache#39076

Uh oh!

Conversation

zzzxl1993 commented Aug 8, 2024• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Proposed changes

Uh oh!

doris-robot commented Aug 8, 2024

Uh oh!

github-actionsBot commented Aug 8, 2024

Uh oh!

zzzxl1993 commented Aug 8, 2024

Uh oh!

github-actionsBot commented Aug 8, 2024

Uh oh!

github-actionsBot commented Aug 8, 2024

Uh oh!

github-actionsBot commented Aug 8, 2024

Uh oh!

doris-robot commented Aug 8, 2024

Uh oh!

github-actionsBot commented Aug 9, 2024

Uh oh!

github-actionsBot commented Aug 9, 2024

Uh oh!

zzzxl1993 commented Aug 9, 2024

Uh oh!

github-actionsBot commented Aug 12, 2024

Uh oh!

zzzxl1993 commented Aug 13, 2024

Uh oh!

github-actionsBot commented Aug 13, 2024

Uh oh!

doris-robot commented Aug 13, 2024

Uh oh!

doris-robot commented Aug 13, 2024

Uh oh!

doris-robot commented Aug 13, 2024

Uh oh!

airborne12 left a comment

Choose a reason for hiding this comment

Uh oh!

github-actionsBot commented Aug 13, 2024

Uh oh!

github-actionsBot commented Aug 13, 2024

Uh oh!

qidaye left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

zzzxl1993 commented Aug 8, 2024•
edited
Loading