Skip to content

[fix](index compaction) fix fd leak and mem leak while index compaction - #41915

Merged
cambyzju merged 1 commit into
apache:masterfrom
cambyzju:fix-master-1015
Oct 21, 2024
Merged

[fix](index compaction) fix fd leak and mem leak while index compaction#41915
cambyzju merged 1 commit into
apache:masterfrom
cambyzju:fix-master-1015

Conversation

@cambyzju

@cambyzjucambyzju commented Oct 15, 2024•
edited
Loading

Copy link
Copy Markdown
Contributor

if DORIS_TRY(inverted_index_file_readers[src_segment_id]->open(index_meta)) failed and return,
then lucene::store::Directory* stay in src_index_dirs will got fd leak and memory leak.

@doris-robot

doris-robot commented Oct 15, 2024

Copy link
Copy Markdown

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@github-actions

github-actionsBot commented Oct 15, 2024

Copy link
Copy Markdown
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@cambyzju

cambyzju commented Oct 15, 2024

Copy link
Copy Markdown
ContributorAuthor

run buildall

@doris-robot

doris-robot commented Oct 15, 2024

Copy link
Copy Markdown

TeamCity be ut coverage result:
Function Coverage: 37.46% (9707/25916)
Line Coverage: 28.72% (80569/280501)
Region Coverage: 28.17% (41686/147970)
Branch Coverage: 24.75% (21199/85636)
Coverage Report: http://coverage.selectdb-in.cc/coverage/ee28396e4272aaf87cbccf286e42c46a91ca65e8_ee28396e4272aaf87cbccf286e42c46a91ca65e8/report/index.html

@doris-robot

doris-robot commented Oct 15, 2024

Copy link
Copy Markdown
TPC-H: Total hot run time: 41362 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit ee28396e4272aaf87cbccf286e42c46a91ca65e8, data reload: false
------ Round 1 ----------------------------------
q1	17584	7365	7288	7288
q2	2014	300	285	285
q3	12045	1065	1235	1065
q4	10586	866	745	745
q5	7764	3087	3095	3087
q6	234	148	148	148
q7	1025	609	623	609
q8	9350	2008	1917	1917
q9	6536	6474	6448	6448
q10	7055	2452	2485	2452
q11	454	242	234	234
q12	411	219	229	219
q13	17770	3012	3031	3012
q14	244	208	219	208
q15	584	520	509	509
q16	647	586	593	586
q17	984	572	495	495
q18	7279	6714	6708	6708
q19	1348	987	1024	987
q20	492	181	187	181
q21	4177	3180	3273	3180
q22	1128	999	1004	999
Total cold run time: 109711 ms
Total hot run time: 41362 ms
----- Round 2, with runtime_filter_mode=off -----
q1	7291	7219	7228	7219
q2	323	234	228	228
q3	3056	2981	2970	2970
q4	2090	1812	1886	1812
q5	5897	5877	5977	5877
q6	236	152	149	149
q7	2368	1886	1879	1879
q8	3437	3475	3510	3475
q9	8947	8940	8927	8927
q10	3591	3574	3558	3558
q11	598	483	496	483
q12	904	634	635	634
q13	10664	3240	3209	3209
q14	315	289	293	289
q15	597	521	530	521
q16	705	645	633	633
q17	1815	1630	1600	1600
q18	8342	7724	7757	7724
q19	1710	1385	1463	1385
q20	2136	1883	1858	1858
q21	5760	5430	5404	5404
q22	1166	1079	1031	1031
Total cold run time: 71948 ms
Total hot run time: 60865 ms

@doris-robot

doris-robot commented Oct 15, 2024

Copy link
Copy Markdown
TPC-DS: Total hot run time: 192756 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit ee28396e4272aaf87cbccf286e42c46a91ca65e8, data reload: false
query1	831	391	391	391
query2	6274	2106	2055	2055
query3	8689	198	206	198
query4	34444	23794	23524	23524
query5	3511	483	459	459
query6	277	172	168	168
query7	4210	302	294	294
query8	303	248	236	236
query9	9568	2749	2736	2736
query10	467	276	263	263
query11	17878	15323	15183	15183
query12	161	103	105	103
query13	1568	429	435	429
query14	9662	7726	7618	7618
query15	262	173	185	173
query16	7522	515	489	489
query17	1656	602	577	577
query18	1970	316	326	316
query19	371	156	160	156
query20	124	114	111	111
query21	213	111	110	110
query22	4873	4668	4537	4537
query23	35140	34267	35780	34267
query24	11137	2775	2822	2775
query25	546	414	401	401
query26	734	173	183	173
query27	2341	284	291	284
query28	7523	2448	2430	2430
query29	668	424	420	420
query30	262	156	157	156
query31	1069	811	828	811
query32	103	55	56	55
query33	759	299	292	292
query34	933	501	511	501
query35	894	755	753	753
query36	1109	946	976	946
query37	149	93	93	93
query38	4029	3870	3941	3870
query39	1450	1437	1402	1402
query40	209	99	98	98
query41	48	45	48	45
query42	128	103	101	101
query43	533	478	479	478
query44	1277	820	830	820
query45	197	164	171	164
query46	1164	746	704	704
query47	1962	1862	1869	1862
query48	423	327	341	327
query49	893	445	432	432
query50	842	387	385	385
query51	7144	7079	6980	6980
query52	101	88	85	85
query53	250	180	180	180
query54	1172	423	417	417
query55	81	75	78	75
query56	270	261	282	261
query57	1293	1160	1146	1146
query58	234	257	250	250
query59	3176	3036	2806	2806
query60	285	261	278	261
query61	97	116	101	101
query62	852	683	662	662
query63	221	184	192	184
query64	3914	638	642	638
query65	3288	3226	3223	3223
query66	789	307	305	305
query67	16121	15897	15643	15643
query68	4911	570	575	570
query69	552	284	294	284
query70	1195	1184	1156	1156
query71	365	279	276	276
query72	7427	4037	4002	4002
query73	775	350	359	350
query74	10492	9037	9044	9037
query75	3813	2668	2676	2668
query76	3417	903	1013	903
query77	645	302	293	293
query78	10424	9642	9588	9588
query79	1129	597	592	592
query80	2065	442	448	442
query81	569	242	239	239
query82	309	140	143	140
query83	290	131	132	131
query84	281	72	70	70
query85	1393	289	279	279
query86	388	285	302	285
query87	4663	4362	4298	4298
query88	3041	2244	2175	2175
query89	386	294	282	282
query90	2198	191	204	191
query91	155	99	100	99
query92	64	47	48	47
query93	1060	541	551	541
query94	1058	286	294	286
query95	355	245	247	245
query96	616	282	276	276
query97	3268	3142	3133	3133
query98	222	202	189	189
query99	1585	1306	1301	1301
Total cold run time: 299755 ms
Total hot run time: 192756 ms

@doris-robot

doris-robot commented Oct 15, 2024

Copy link
Copy Markdown
ClickBench: Total hot run time: 33.22 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit ee28396e4272aaf87cbccf286e42c46a91ca65e8, data reload: false
query1	0.04	0.03	0.03
query2	0.07	0.03	0.02
query3	0.23	0.06	0.06
query4	1.64	0.11	0.10
query5	0.50	0.50	0.49
query6	1.12	0.72	0.73
query7	0.02	0.02	0.02
query8	0.03	0.05	0.03
query9	0.57	0.49	0.49
query10	0.55	0.54	0.54
query11	0.15	0.11	0.11
query12	0.14	0.10	0.11
query13	0.62	0.60	0.60
query14	2.72	2.70	2.72
query15	0.90	0.84	0.85
query16	0.38	0.37	0.37
query17	1.00	1.05	1.01
query18	0.20	0.20	0.19
query19	1.93	1.83	2.01
query20	0.02	0.01	0.00
query21	15.35	0.57	0.58
query22	2.84	2.17	2.02
query23	17.05	0.89	0.84
query24	2.89	1.27	1.67
query25	0.22	0.11	0.12
query26	0.46	0.13	0.15
query27	0.05	0.04	0.04
query28	9.98	1.10	1.07
query29	12.54	3.26	3.25
query30	0.24	0.06	0.06
query31	2.86	0.38	0.37
query32	3.31	0.46	0.45
query33	3.04	3.01	3.04
query34	17.13	4.48	4.49
query35	4.51	4.53	4.52
query36	0.67	0.49	0.48
query37	0.08	0.06	0.06
query38	0.05	0.04	0.03
query39	0.04	0.02	0.03
query40	0.16	0.13	0.12
query41	0.08	0.02	0.03
query42	0.03	0.02	0.02
query43	0.04	0.03	0.03
Total cold run time: 106.45 s
Total hot run time: 33.22 s

qidaye
qidaye reviewed Oct 17, 2024
View reviewed changes
Comment threadbe/src/olap/rowset/segment_v2/inverted_index_compaction.cpp
qidaye
qidaye approved these changes Oct 18, 2024
View reviewed changes

@qidayeqidaye left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actionsgithub-actionsBot added the approved Indicates a PR has been approved by one committer. label Oct 18, 2024
@github-actions

github-actionsBot commented Oct 18, 2024

Copy link
Copy Markdown
Contributor

PR approved by at least one committer and no changes requested.

@github-actions

github-actionsBot commented Oct 18, 2024

Copy link
Copy Markdown
Contributor

PR approved by anyone and no changes requested.

lide-reed
lide-reed approved these changes Oct 21, 2024
View reviewed changes

@lide-reedlide-reed left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@cambyzju
cambyzju merged commit f65223f into apache:masterOct 21, 2024
cambyzju added a commit to cambyzju/incubator-doris that referenced this pull request Oct 21, 2024
@cambyzju
[fix](index compaction) fix fd leak and mem leak while index compacti…
b31bf06
…on (apache#41915)
@cambyzjucambyzju mentioned this pull request Oct 21, 2024
Merged
cambyzju added a commit to cambyzju/incubator-doris that referenced this pull request Oct 22, 2024
@cambyzju
[fix](index compaction) fix fd leak and mem leak while index compacti…
7599974
…on (apache#41915)
lide-reed pushed a commit that referenced this pull request Oct 24, 2024
@cambyzju
[fix](index compaction) fix fd leak and mem leak while index compacti…
32b13b2
…on (#41915) (#42171)
pick #41915 to branch-2.1
airborne12 pushed a commit to airborne12/apache-doris that referenced this pull request Oct 24, 2024
@cambyzju@airborne12
[fix](index compaction) fix fd leak and mem leak while index compacti…
66f58b6
…on (apache#41915)
@airborne12airborne12 mentioned this pull request Oct 24, 2024
Merged
airborne12 added a commit that referenced this pull request Oct 24, 2024
@airborne12@cambyzju
[fix](index compaction) fix fd leak and mem leak while index compaction
b39e2e1
#41915 (#42374)
cherry pick from #41915
Co-authored-by: camby <cambyzhu@tencent.com>
@yiguoleiyiguolei mentioned this pull request Nov 6, 2024
Closed
@yiguoleiyiguolei mentioned this pull request Jan 19, 2025
Closed
HappenLee pushed a commit to HappenLee/incubator-doris that referenced this pull request Apr 24, 2026
@cambyzju@HappenLee
[fix](index compaction) fix fd leak and mem leak while index compacti…
9e79b48
…on (apache#41915)
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@lide-reedlide-reedlide-reed approved these changes
@qidayeqidayeqidaye approved these changes
@airborne12airborne12Awaiting requested review from airborne12

Assignees

No one assigned

Labels

approvedIndicates a PR has been approved by one committer.dev/2.1.7-mergeddev/3.0.3-mergedreviewed

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

6 participants

@cambyzju@doris-robot@lide-reed@qidaye@airborne12@wm1581066