Skip to content

[enhancement](delete-pred) enable delete sub predicate v2 for compaction - #35859

Merged
dataroaring merged 1 commit into
apache:masterfrom
Hastyshell:compaction-use-delete-subpredicate-v2
Jun 5, 2024
Merged

[enhancement](delete-pred) enable delete sub predicate v2 for compaction#35859
dataroaring merged 1 commit into
apache:masterfrom
Hastyshell:compaction-use-delete-subpredicate-v2

Conversation

@Hastyshell

@HastyshellHastyshell commented Jun 4, 2024

Copy link
Copy Markdown
Collaborator

Proposed changes

This PR enable elete sub predicate v2 for compaction, and legacy version of delete predicate will be processed in the original way.

@doris-robot

doris-robot commented Jun 4, 2024

Copy link
Copy Markdown

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@Hastyshell

Hastyshell commented Jun 4, 2024

Copy link
Copy Markdown
CollaboratorAuthor

run buildall

github-actions[bot]
github-actionsBot reviewed Jun 4, 2024
View reviewed changes

@github-actionsgithub-actionsBot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

clang-tidy made some suggestions

Comment threadbe/src/olap/delete_handler.h
@@ -18,12 +18,9 @@
#pragma once

#include <butil/macros.h>

@github-actionsgithub-actionsBotJun 4, 2024

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

warning: 'butil/macros.h' file not found [clang-diagnostic-error]

#include<butil/macros.h>
^

@github-actions

github-actionsBot commented Jun 4, 2024

Copy link
Copy Markdown
Contributor

PR approved by anyone and no changes requested.

@doris-robot

doris-robot commented Jun 4, 2024

Copy link
Copy Markdown

TeamCity be ut coverage result:
Function Coverage: 37.40% (9284/24823)
Line Coverage: 28.73% (76025/264594)
Region Coverage: 28.12% (39389/140084)
Branch Coverage: 24.53% (19926/81224)
Coverage Report: http://coverage.selectdb-in.cc/coverage/f5cf02e64d63f5c7e712921fe42d308eee3e49a0_f5cf02e64d63f5c7e712921fe42d308eee3e49a0/report/index.html

@doris-robot

doris-robot commented Jun 4, 2024

Copy link
Copy Markdown
TPC-H: Total hot run time: 41306 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit f5cf02e64d63f5c7e712921fe42d308eee3e49a0, data reload: false
------ Round 1 ----------------------------------
q1	17613	4918	4339	4339
q2	2026	194	190	190
q3	10527	1201	1122	1122
q4	10212	785	831	785
q5	7486	2788	2771	2771
q6	232	139	139	139
q7	976	646	620	620
q8	9221	2192	2151	2151
q9	9351	6759	6845	6759
q10	9720	3969	3906	3906
q11	449	249	245	245
q12	429	233	250	233
q13	18302	3180	3212	3180
q14	276	218	209	209
q15	530	488	485	485
q16	478	396	400	396
q17	1044	781	643	643
q18	8503	7916	7890	7890
q19	6594	1416	1511	1416
q20	671	331	321	321
q21	5181	3171	4005	3171
q22	412	335	339	335
Total cold run time: 120233 ms
Total hot run time: 41306 ms
----- Round 2, with runtime_filter_mode=off -----
q1	4620	4464	4479	4464
q2	386	271	267	267
q3	3189	2909	2973	2909
q4	1896	1617	1695	1617
q5	5472	5552	5508	5508
q6	233	127	129	127
q7	2226	1820	1797	1797
q8	3278	3388	3399	3388
q9	8714	8688	8760	8688
q10	4095	3893	3772	3772
q11	575	491	481	481
q12	792	605	595	595
q13	15985	3165	3059	3059
q14	317	275	286	275
q15	540	485	495	485
q16	472	422	428	422
q17	1822	1522	1494	1494
q18	8285	7513	7458	7458
q19	1834	1537	1504	1504
q20	3040	1767	1796	1767
q21	4950	4678	4664	4664
q22	634	545	521	521
Total cold run time: 73355 ms
Total hot run time: 55262 ms

@doris-robot

doris-robot commented Jun 4, 2024

Copy link
Copy Markdown
TPC-DS: Total hot run time: 172872 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit f5cf02e64d63f5c7e712921fe42d308eee3e49a0, data reload: false
query1	929	392	370	370
query2	6438	2450	2390	2390
query3	6636	204	207	204
query4	20639	17262	17270	17262
query5	4107	451	456	451
query6	237	158	168	158
query7	4601	291	286	286
query8	322	296	278	278
query9	8422	2429	2389	2389
query10	422	290	276	276
query11	10632	9993	9996	9993
query12	136	86	89	86
query13	1621	353	360	353
query14	10272	6993	7590	6993
query15	241	181	184	181
query16	7781	263	255	255
query17	1298	513	522	513
query18	1910	298	282	282
query19	206	163	165	163
query20	94	90	86	86
query21	205	154	135	135
query22	4430	4106	3976	3976
query23	33728	33116	33099	33099
query24	11348	2806	2755	2755
query25	632	374	359	359
query26	1520	153	150	150
query27	3005	310	321	310
query28	7297	2045	2055	2045
query29	980	619	596	596
query30	252	146	150	146
query31	963	745	729	729
query32	93	54	53	53
query33	745	270	288	270
query34	965	474	495	474
query35	744	624	603	603
query36	1104	925	930	925
query37	151	66	68	66
query38	2857	2732	2714	2714
query39	867	797	783	783
query40	218	124	121	121
query41	53	51	52	51
query42	120	95	96	95
query43	592	559	558	558
query44	1193	735	742	735
query45	200	165	172	165
query46	1082	734	718	718
query47	1826	1763	1783	1763
query48	361	292	298	292
query49	923	393	400	393
query50	777	387	388	387
query51	6981	6745	6748	6745
query52	105	93	89	89
query53	357	285	315	285
query54	976	437	430	430
query55	79	71	72	71
query56	285	256	255	255
query57	1154	1039	1035	1035
query58	259	260	242	242
query59	3608	3155	3299	3155
query60	288	261	266	261
query61	90	83	85	83
query62	632	436	452	436
query63	324	287	287	287
query64	9956	2315	1818	1818
query65	3216	3103	3194	3103
query66	1343	342	344	342
query67	15525	15292	15010	15010
query68	4467	536	537	536
query69	471	326	335	326
query70	1082	1128	1082	1082
query71	413	299	289	289
query72	7227	5573	5996	5573
query73	757	321	321	321
query74	5917	5574	5415	5415
query75	3350	2653	2674	2653
query76	2289	901	945	901
query77	418	302	299	299
query78	10627	9909	9778	9778
query79	2089	511	524	511
query80	2675	483	470	470
query81	555	219	230	219
query82	765	109	104	104
query83	305	175	175	175
query84	266	92	87	87
query85	2205	319	309	309
query86	495	316	310	310
query87	3270	3117	3079	3079
query88	4062	2494	2363	2363
query89	485	397	379	379
query90	1958	182	182	182
query91	125	99	93	93
query92	61	48	50	48
query93	2345	503	494	494
query94	1374	180	182	180
query95	404	303	306	303
query96	604	264	261	261
query97	3180	3005	3009	3005
query98	246	216	208	208
query99	1205	837	844	837
Total cold run time: 277045 ms
Total hot run time: 172872 ms

@doris-robot

doris-robot commented Jun 4, 2024

Copy link
Copy Markdown
ClickBench: Total hot run time: 30.53 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit f5cf02e64d63f5c7e712921fe42d308eee3e49a0, data reload: false
query1	0.04	0.03	0.03
query2	0.09	0.04	0.04
query3	0.22	0.06	0.05
query4	1.67	0.07	0.08
query5	0.48	0.50	0.48
query6	1.13	0.72	0.72
query7	0.02	0.01	0.02
query8	0.05	0.04	0.04
query9	0.54	0.48	0.47
query10	0.53	0.54	0.53
query11	0.17	0.11	0.11
query12	0.16	0.12	0.12
query13	0.58	0.58	0.60
query14	0.76	0.78	0.77
query15	0.82	0.81	0.80
query16	0.38	0.34	0.38
query17	0.95	0.94	0.98
query18	0.20	0.28	0.21
query19	1.86	1.72	1.70
query20	0.02	0.01	0.00
query21	15.72	0.64	0.65
query22	3.97	7.26	2.02
query23	18.30	1.39	1.20
query24	1.68	0.25	0.24
query25	0.13	0.08	0.08
query26	0.27	0.18	0.17
query27	0.08	0.08	0.07
query28	13.44	1.12	1.09
query29	13.05	3.29	3.26
query30	0.25	0.05	0.05
query31	2.88	0.40	0.39
query32	3.25	0.49	0.46
query33	2.85	2.92	2.90
query34	16.92	4.43	4.39
query35	4.47	4.65	4.51
query36	0.65	0.45	0.46
query37	0.18	0.15	0.15
query38	0.16	0.15	0.14
query39	0.05	0.04	0.03
query40	0.16	0.15	0.17
query41	0.10	0.05	0.04
query42	0.05	0.05	0.05
query43	0.04	0.04	0.04
Total cold run time: 109.32 s
Total hot run time: 30.53 s

dataroaring
dataroaring approved these changes Jun 5, 2024
View reviewed changes

@dataroaringdataroaring left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@dataroaring
dataroaring merged commit 2a57976 into apache:masterJun 5, 2024
@github-actionsgithub-actionsBot added the approved Indicates a PR has been approved by one committer. label Jun 5, 2024
@github-actions

github-actionsBot commented Jun 5, 2024

Copy link
Copy Markdown
Contributor

PR approved by at least one committer and no changes requested.

Hastyshell added a commit to Hastyshell/doris that referenced this pull request Jun 5, 2024
@Hastyshell
[enhancement](delete-pred) enable delete sub predicate v2 for compact…
7ff68dd
…ion (apache#35859)
This PR enable elete sub predicate v2 for compaction, and legacy version
of delete predicate will be processed in the original way.
dataroaring pushed a commit that referenced this pull request Jun 5, 2024
@Hastyshell
[enhancement](delete-pred) enable delete sub predicate v2 for compact…
fdd87fe
…ion (#35859) (#35895)
## Proposed changes
This PR enable `delete sub predicate v2` for compaction, and legacy
version of delete predicate will be processed in the original way.
dataroaring pushed a commit that referenced this pull request Jun 7, 2024
@Hastyshell@dataroaring
[enhancement](delete-pred) enable delete sub predicate v2 for compact…
38b31a2
…ion (#35859)
This PR enable elete sub predicate v2 for compaction, and legacy version
of delete predicate will be processed in the original way.
HappenLee pushed a commit to HappenLee/incubator-doris that referenced this pull request Apr 24, 2026
@Hastyshell@HappenLee
[enhancement](delete-pred) enable delete sub predicate v2 for compact…
2637d25
…ion (apache#35859)
This PR enable elete sub predicate v2 for compaction, and legacy version
of delete predicate will be processed in the original way.
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@github-actionsgithub-actions[bot]github-actions[bot] left review comments
@gavinchougavinchougavinchou approved these changes
@dataroaringdataroaringdataroaring approved these changes

Assignees

No one assigned

Labels

approvedIndicates a PR has been approved by one committer.dev/2.1.4-mergeddev/3.0.0-mergedreviewed

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@Hastyshell@doris-robot@gavinchou@dataroaring@morningman