Skip to content

[fix](index compaction)Skip writing terms with a doc frequency of 0 (#43113) - #43225

Merged
airborne12 merged 3 commits into
apache:branch-2.0from
qidaye:pick_43113_2.0
Nov 5, 2024
Merged

[fix](index compaction)Skip writing terms with a doc frequency of 0 (#43113)#43225
airborne12 merged 3 commits into
apache:branch-2.0from
qidaye:pick_43113_2.0

Conversation

@qidaye

@qidayeqidaye commented Nov 4, 2024

Copy link
Copy Markdown
Contributor

@doris-robot

doris-robot commented Nov 4, 2024

Copy link
Copy Markdown

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@qidaye

qidaye commented Nov 4, 2024

Copy link
Copy Markdown
ContributorAuthor

run buildall

github-actions[bot]
github-actionsBot reviewed Nov 4, 2024
View reviewed changes

@github-actionsgithub-actionsBot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

clang-tidy made some suggestions

Comment threadbe/test/olap/rowset/segment_v2/inverted_index/index_compaction_with_deleted_term.cpp
// specific language governing permissions and limitations
// under the License.

#include <gtest/gtest.h>

@github-actionsgithub-actionsBotNov 4, 2024

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

warning: 'gtest/gtest.h' file not found [clang-diagnostic-error]

#include<gtest/gtest.h>
^

@doris-robot

doris-robot commented Nov 4, 2024

Copy link
Copy Markdown
TPC-H: Total hot run time: 48970 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 1dd51df91e3410ca214fcf9475bcb05495550232, data reload: false
------ Round 1 ----------------------------------
q1	17931	4366	4281	4281
q2	2067	156	141	141
q3	10462	1884	1901	1884
q4	10319	1274	1324	1274
q5	8474	3898	3851	3851
q6	226	121	124	121
q7	2020	1649	1616	1616
q8	9286	2724	2696	2696
q9	10217	9895	9806	9806
q10	8640	3519	3533	3519
q11	420	242	235	235
q12	472	302	302	302
q13	18340	3969	4020	3969
q14	355	334	320	320
q15	517	450	448	448
q16	535	462	467	462
q17	1127	946	949	946
q18	7265	6810	6874	6810
q19	1684	1555	1529	1529
q20	536	308	295	295
q21	4432	4119	4088	4088
q22	499	377	396	377
Total cold run time: 115824 ms
Total hot run time: 48970 ms
----- Round 2, with runtime_filter_mode=off -----
q1	4311	4275	4307	4275
q2	321	228	220	220
q3	4162	4136	4136	4136
q4	2743	2759	2730	2730
q5	7151	7105	7101	7101
q6	239	119	123	119
q7	3250	2860	2793	2793
q8	4413	4461	4505	4461
q9	13651	13433	13568	13433
q10	4268	4258	4260	4258
q11	798	671	679	671
q12	1010	866	859	859
q13	6293	3758	3759	3758
q14	445	429	423	423
q15	505	462	460	460
q16	642	618	579	579
q17	3746	3818	3951	3818
q18	8821	8684	8802	8684
q19	1726	1662	1634	1634
q20	2389	2105	2119	2105
q21	8548	8483	8558	8483
q22	1031	915	966	915
Total cold run time: 80463 ms
Total hot run time: 75915 ms

@doris-robot

doris-robot commented Nov 4, 2024

Copy link
Copy Markdown
TPC-DS: Total hot run time: 212440 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 1dd51df91e3410ca214fcf9475bcb05495550232, data reload: false
query1	941	390	407	390
query2	6562	2153	2134	2134
query3	6915	201	205	201
query4	23716	21180	21746	21180
query5	19744	6540	6527	6527
query6	283	218	239	218
query7	4160	298	306	298
query8	261	256	276	256
query9	3079	2664	2592	2592
query10	412	318	314	314
query11	15461	14879	15532	14879
query12	136	74	76	74
query13	1047	446	446	446
query14	17031	13626	13315	13315
query15	396	231	224	224
query16	6493	272	259	259
query17	1765	890	861	861
query18	900	317	316	316
query19	209	150	149	149
query20	106	103	103	103
query21	184	94	94	94
query22	5141	4991	4757	4757
query23	34235	33574	33334	33334
query24	7943	6302	6331	6302
query25	528	434	432	432
query26	1275	171	161	161
query27	2519	292	297	292
query28	6094	2231	2203	2203
query29	2836	2852	2882	2852
query30	244	167	164	164
query31	954	746	726	726
query32	70	64	58	58
query33	442	273	264	264
query34	872	472	492	472
query35	1128	920	937	920
query36	1365	1253	1142	1142
query37	176	60	62	60
query38	3090	2923	2869	2869
query39	1373	1311	1332	1311
query40	308	98	94	94
query41	42	38	36	36
query42	87	92	89	89
query43	619	592	576	576
query44	1161	724	726	724
query45	247	231	229	229
query46	1230	974	971	971
query47	1831	1866	1705	1705
query48	516	404	415	404
query49	670	384	370	370
query50	860	626	621	621
query51	4723	4716	4656	4656
query52	87	88	75	75
query53	230	180	190	180
query54	2652	2468	2511	2468
query55	83	86	83	83
query56	229	187	209	187
query57	1232	1265	1091	1091
query58	219	218	192	192
query59	3488	3210	3222	3210
query60	220	208	200	200
query61	96	96	99	96
query62	870	444	522	444
query63	196	177	178	177
query64	3574	1589	1434	1434
query65	3640	3585	3560	3560
query66	761	436	436	436
query67	15617	16968	16554	16554
query68	9872	670	634	634
query69	503	258	270	258
query70	1698	1342	1570	1342
query71	415	303	306	303
query72	6795	4897	4811	4811
query73	772	310	318	310
query74	6298	5864	5788	5788
query75	5359	3620	3650	3620
query76	5616	1131	1197	1131
query77	959	255	251	251
query78	12452	11705	11568	11568
query79	11467	635	647	635
query80	1983	385	388	385
query81	498	239	233	233
query82	1641	99	98	98
query83	165	132	134	132
query84	258	69	72	69
query85	881	320	310	310
query86	329	314	296	296
query87	3226	2993	3069	2993
query88	5222	2297	2280	2280
query89	491	295	295	295
query90	2003	217	215	215
query91	157	126	129	126
query92	61	53	55	53
query93	7354	552	584	552
query94	744	212	211	211
query95	1933	1753	1910	1753
query96	650	322	324	322
query97	6471	6356	6438	6356
query98	225	208	208	208
query99	3152	945	891	891
Total cold run time: 325224 ms
Total hot run time: 212440 ms

@doris-robot

doris-robot commented Nov 4, 2024

Copy link
Copy Markdown
ClickBench: Total hot run time: 30.4 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 1dd51df91e3410ca214fcf9475bcb05495550232, data reload: false
query1	0.02	0.02	0.03
query2	0.07	0.02	0.02
query3	0.24	0.06	0.06
query4	1.81	0.11	0.10
query5	0.54	0.53	0.52
query6	1.26	0.61	0.62
query7	0.02	0.01	0.01
query8	0.04	0.02	0.02
query9	0.53	0.50	0.49
query10	0.55	0.54	0.54
query11	0.12	0.09	0.09
query12	0.12	0.10	0.10
query13	0.62	0.62	0.61
query14	0.77	0.79	0.82
query15	0.79	0.76	0.75
query16	0.38	0.36	0.36
query17	0.99	1.01	1.01
query18	0.24	0.26	0.23
query19	1.94	1.87	1.86
query20	0.02	0.01	0.01
query21	15.46	0.54	0.56
query22	2.28	2.82	1.45
query23	16.95	0.98	0.87
query24	6.69	0.79	1.55
query25	0.57	0.12	0.05
query26	0.74	0.16	0.15
query27	0.04	0.04	0.05
query28	6.51	0.75	0.71
query29	12.62	2.31	2.25
query30	0.59	0.53	0.52
query31	2.83	0.38	0.38
query32	3.36	0.50	0.49
query33	3.06	3.08	3.05
query34	15.26	4.78	4.80
query35	4.84	4.85	4.86
query36	1.08	1.05	1.03
query37	0.06	0.04	0.05
query38	0.04	0.02	0.02
query39	0.02	0.02	0.02
query40	0.17	0.14	0.15
query41	0.07	0.02	0.01
query42	0.02	0.02	0.01
query43	0.02	0.01	0.02
Total cold run time: 104.35 s
Total hot run time: 30.4 s

@doris-robot

doris-robot commented Nov 4, 2024

Copy link
Copy Markdown

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit 1dd51df91e3410ca214fcf9475bcb05495550232 with default session variables
Stream load json: 20 seconds loaded 2358488459 Bytes, about 112 MB/s
Stream load orc: 58 seconds loaded 1101869774 Bytes, about 18 MB/s
Stream load parquet: 31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select: 21.7 seconds inserted 10000000 Rows, about 460K ops/s

@doris-robot

doris-robot commented Nov 4, 2024

Copy link
Copy Markdown

TeamCity be ut coverage result:
Function Coverage: 38.55% (8308/21553)
Line Coverage: 30.25% (68678/227052)
Region Coverage: 29.68% (35372/119186)
Branch Coverage: 25.45% (18194/71492)
Coverage Report: http://coverage.selectdb-in.cc/coverage/1dd51df91e3410ca214fcf9475bcb05495550232_1dd51df91e3410ca214fcf9475bcb05495550232/report/index.html

@airborne12
airborne12 merged commit d700134 into apache:branch-2.0Nov 5, 2024
@qidaye
qidaye deleted the pick_43113_2.0 branch November 5, 2024 06:42
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@github-actionsgithub-actions[bot]github-actions[bot] left review comments

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@qidaye@doris-robot@airborne12