Skip to content

[fix](segcompaction) fix segcompaction file not found when inverted index storage format is v1 - #46054

Merged
airborne12 merged 1 commit into
apache:masterfrom
airborne12:fix-segcompaction
Dec 30, 2024
Merged

[fix](segcompaction) fix segcompaction file not found when inverted index storage format is v1#46054
airborne12 merged 1 commit into
apache:masterfrom
airborne12:fix-segcompaction

Conversation

@airborne12

@airborne12airborne12 commented Dec 26, 2024•
edited
Loading

Copy link
Copy Markdown
Member

What problem does this PR solve?

Issue Number: close #xxx

Related PR: #41625

Problem Summary:
Wrong inverted index file path for inverted index storage format v1 when making segment compaction.

Release note

None

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason

  • Behavior changed:

    • No.
    • Yes.

  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@Thearas

Thearas commented Dec 26, 2024

Copy link
Copy Markdown
Contributor

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@airborne12

airborne12 commented Dec 27, 2024

Copy link
Copy Markdown
MemberAuthor

run buildall

1 similar comment
@airborne12

airborne12 commented Dec 27, 2024

Copy link
Copy Markdown
MemberAuthor

run buildall

@doris-robot

doris-robot commented Dec 27, 2024

Copy link
Copy Markdown
TPC-H: Total hot run time: 32686 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 980bc17044059bd01c871591472e401a3fb480dc, data reload: false
------ Round 1 ----------------------------------
q1	17563	6151	5988	5988
q2	2050	291	166	166
q3	10431	1237	743	743
q4	10212	864	448	448
q5	7526	2200	1980	1980
q6	211	186	152	152
q7	907	763	614	614
q8	9227	1365	1194	1194
q9	5244	4903	4984	4903
q10	6750	2316	1870	1870
q11	481	286	244	244
q12	353	364	227	227
q13	17746	3606	2973	2973
q14	234	237	217	217
q15	564	505	505	505
q16	625	625	626	625
q17	587	858	330	330
q18	7098	6652	6464	6464
q19	2515	966	562	562
q20	308	314	187	187
q21	2918	2232	1984	1984
q22	373	340	310	310
Total cold run time: 103923 ms
Total hot run time: 32686 ms
----- Round 2, with runtime_filter_mode=off -----
q1	6357	6288	6306	6288
q2	236	325	237	237
q3	2260	2668	2343	2343
q4	1557	1904	1410	1410
q5	4382	4813	4927	4813
q6	183	182	144	144
q7	2078	1991	1848	1848
q8	2663	2878	2706	2706
q9	7363	7299	7413	7299
q10	3142	3453	2944	2944
q11	602	522	499	499
q12	680	788	629	629
q13	3603	4064	3321	3321
q14	308	319	323	319
q15	594	517	539	517
q16	667	707	670	670
q17	1305	1812	1289	1289
q18	8174	7508	7449	7449
q19	1860	1421	1069	1069
q20	2044	2094	1923	1923
q21	5820	5209	5037	5037
q22	604	625	551	551
Total cold run time: 56482 ms
Total hot run time: 53305 ms

@doris-robot

doris-robot commented Dec 27, 2024

Copy link
Copy Markdown
TPC-DS: Total hot run time: 198462 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 980bc17044059bd01c871591472e401a3fb480dc, data reload: false
query1	1308	995	923	923
query2	6525	2399	2324	2324
query3	11017	4775	4785	4775
query4	32928	24324	23981	23981
query5	4962	654	468	468
query6	293	210	200	200
query7	4004	497	312	312
query8	307	260	246	246
query9	9644	2767	2767	2767
query10	507	306	277	277
query11	18201	15299	15548	15299
query12	175	128	102	102
query13	1672	520	407	407
query14	11140	7674	7223	7223
query15	242	213	192	192
query16	8067	600	421	421
query17	1491	775	577	577
query18	2130	415	340	340
query19	194	200	173	173
query20	156	115	116	115
query21	204	128	116	116
query22	4747	4731	4431	4431
query23	34418	33652	33753	33652
query24	6435	2378	2332	2332
query25	501	462	400	400
query26	833	284	151	151
query27	2224	468	328	328
query28	5524	2509	2494	2494
query29	666	562	429	429
query30	208	179	159	159
query31	983	929	849	849
query32	96	58	59	58
query33	506	378	307	307
query34	821	860	554	554
query35	834	834	760	760
query36	1037	1082	973	973
query37	125	107	77	77
query38	4192	4287	4156	4156
query39	1520	1480	1479	1479
query40	227	124	115	115
query41	47	53	48	48
query42	128	108	111	108
query43	530	539	505	505
query44	1414	828	850	828
query45	194	192	182	182
query46	949	1072	696	696
query47	2009	2003	1949	1949
query48	401	429	338	338
query49	747	530	426	426
query50	670	689	466	466
query51	7253	7231	7234	7231
query52	102	115	92	92
query53	230	269	187	187
query54	497	498	413	413
query55	88	82	79	79
query56	296	270	265	265
query57	1263	1240	1173	1173
query58	247	232	245	232
query59	3283	3245	3135	3135
query60	292	288	265	265
query61	117	110	110	110
query62	897	803	801	801
query63	242	204	199	199
query64	3682	1039	683	683
query65	3365	3341	3332	3332
query66	848	410	309	309
query67	16674	15862	15595	15595
query68	10209	763	523	523
query69	489	307	259	259
query70	1282	1160	1137	1137
query71	462	296	261	261
query72	5911	3882	3930	3882
query73	1016	765	358	358
query74	9741	9356	8976	8976
query75	4672	3166	2709	2709
query76	5599	1170	791	791
query77	1043	374	299	299
query78	10518	10395	9264	9264
query79	3035	879	600	600
query80	720	530	430	430
query81	490	279	240	240
query82	607	156	121	121
query83	196	161	147	147
query84	282	100	79	79
query85	782	374	314	314
query86	352	319	307	307
query87	4426	4584	4495	4495
query88	3431	2250	2233	2233
query89	429	334	306	306
query90	1917	191	191	191
query91	137	141	108	108
query92	72	96	51	51
query93	1418	862	530	530
query94	658	395	298	298
query95	335	276	271	271
query96	492	614	273	273
query97	2700	2861	2745	2745
query98	221	204	199	199
query99	1701	1578	1476	1476
Total cold run time: 302872 ms
Total hot run time: 198462 ms

@doris-robot

doris-robot commented Dec 27, 2024

Copy link
Copy Markdown

TeamCity be ut coverage result:
Function Coverage: 38.89% (10121/26022)
Line Coverage: 29.88% (85474/286099)
Region Coverage: 29.03% (43700/150556)
Branch Coverage: 25.56% (22291/87208)
Coverage Report: http://coverage.selectdb-in.cc/coverage/980bc17044059bd01c871591472e401a3fb480dc_980bc17044059bd01c871591472e401a3fb480dc/report/index.html

@doris-robot

doris-robot commented Dec 27, 2024

Copy link
Copy Markdown
ClickBench: Total hot run time: 31.95 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 980bc17044059bd01c871591472e401a3fb480dc, data reload: false
query1	0.03	0.03	0.03
query2	0.07	0.03	0.03
query3	0.23	0.07	0.07
query4	1.61	0.10	0.11
query5	0.41	0.41	0.41
query6	1.15	0.66	0.64
query7	0.02	0.01	0.02
query8	0.04	0.04	0.02
query9	0.58	0.48	0.52
query10	0.55	0.59	0.56
query11	0.15	0.10	0.10
query12	0.14	0.10	0.11
query13	0.61	0.61	0.59
query14	2.73	2.85	2.84
query15	0.91	0.82	0.83
query16	0.38	0.38	0.36
query17	1.07	1.04	1.03
query18	0.22	0.21	0.21
query19	1.88	1.90	1.99
query20	0.02	0.00	0.02
query21	15.37	0.93	0.60
query22	0.75	0.85	0.59
query23	15.30	1.44	0.60
query24	3.05	1.73	1.60
query25	0.17	0.10	0.07
query26	0.23	0.16	0.14
query27	0.05	0.05	0.04
query28	14.30	1.48	1.05
query29	12.54	3.98	3.29
query30	0.25	0.09	0.06
query31	2.84	0.58	0.37
query32	3.23	0.56	0.47
query33	3.12	3.04	3.07
query34	16.70	5.09	4.50
query35	4.50	4.45	4.48
query36	0.66	0.49	0.48
query37	0.10	0.06	0.06
query38	0.04	0.04	0.03
query39	0.04	0.02	0.03
query40	0.17	0.14	0.12
query41	0.08	0.03	0.02
query42	0.04	0.02	0.02
query43	0.04	0.03	0.03
Total cold run time: 106.37 s
Total hot run time: 31.95 s

qidaye
qidaye approved these changes Dec 27, 2024
View reviewed changes

@qidayeqidaye left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actionsgithub-actionsBot added the approved Indicates a PR has been approved by one committer. label Dec 27, 2024
@github-actions

github-actionsBot commented Dec 27, 2024

Copy link
Copy Markdown
Contributor

PR approved by at least one committer and no changes requested.

@github-actions

github-actionsBot commented Dec 27, 2024

Copy link
Copy Markdown
Contributor

PR approved by anyone and no changes requested.

zzzxl1993
zzzxl1993 approved these changes Dec 27, 2024
View reviewed changes

@zzzxl1993zzzxl1993 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@airborne12
airborne12 merged commit 7b75347 into apache:masterDec 30, 2024
@airborne12
airborne12 deleted the fix-segcompaction branch December 30, 2024 02:42
github-actionsBot pushed a commit that referenced this pull request Dec 30, 2024
@airborne12Your Name
[fix](segcompaction) fix segcompaction file not found when inverted i…
92e0656
…ndex storage format is v1 (#46054)
Problem Summary:
Wrong inverted index file path for inverted index storage format v1 when
making segment compaction.
@github-actionsgithub-actionsBot mentioned this pull request Dec 30, 2024
Merged
airborne12 added a commit that referenced this pull request Dec 30, 2024
@github-actions@airborne12
branch-3.0: [fix](segcompaction) fix segcompaction file not found whe…
46d3281
…n inverted index storage format is v1 #46054 (#46142)
Cherry-picked from #46054
Co-authored-by: airborne12 <jiangkai@selectdb.com>
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@qidayeqidayeqidaye approved these changes
@zzzxl1993zzzxl1993zzzxl1993 approved these changes

Assignees

No one assigned

Labels

approvedIndicates a PR has been approved by one committer.dev/3.0.4-mergedreviewed

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@airborne12@Thearas@doris-robot@qidaye@zzzxl1993