[fix](cloud-mow)Full compaction no need to sync delete bitmap by hust-hhb · Pull Request #43337 · apache/doris

hust-hhb · 2024-11-06T07:14:20Z

sync delete bitmap may cost too much time,which may lead to full compaction fail, but full compaction no need to sync it.

doris-robot · 2024-11-06T07:14:26Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

hust-hhb · 2024-11-06T07:14:32Z

run buildall

github-actions

clang-tidy made some suggestions

github-actions · 2024-11-06T07:19:29Z

 }

-Status CloudMetaMgr::sync_tablet_rowsets(CloudTablet* tablet, bool warmup_delta_data) {
+Status CloudMetaMgr::sync_tablet_rowsets(CloudTablet* tablet, bool warmup_delta_data,


warning: function 'sync_tablet_rowsets' exceeds recommended size/complexity thresholds [readability-function-size]
Status CloudMetaMgr::sync_tablet_rowsets(CloudTablet* tablet, bool warmup_delta_data, ^
Additional context
be/src/cloud/cloud_meta_mgr.cpp:386: 181 lines including whitespace and comments (threshold 80)
Status CloudMetaMgr::sync_tablet_rowsets(CloudTablet* tablet, bool warmup_delta_data, ^

hust-hhb · 2024-11-06T07:20:22Z

run buildall

github-actions · 2024-11-06T07:59:30Z

PR approved by at least one committer and no changes requested.

github-actions · 2024-11-06T07:59:33Z

PR approved by anyone and no changes requested.

doris-robot · 2024-11-06T08:02:18Z

TeamCity be ut coverage result:
Function Coverage: 37.89% (9847/25989)
Line Coverage: 29.04% (81868/281904)
Region Coverage: 28.27% (42177/149181)
Branch Coverage: 24.85% (21399/86106)
Coverage Report: http://coverage.selectdb-in.cc/coverage/35b58b30654b8f4a0b87668407395a983c99a68c_35b58b30654b8f4a0b87668407395a983c99a68c/report/index.html

doris-robot · 2024-11-06T08:04:04Z

TPC-H: Total hot run time: 41542 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 35b58b30654b8f4a0b87668407395a983c99a68c, data reload: false
------ Round 1 ----------------------------------
q1	17572	7491	7324	7324
q2	2048	176	169	169
q3	10833	1102	1154	1102
q4	10563	878	896	878
q5	7738	3097	3063	3063
q6	244	147	150	147
q7	1036	612	614	612
q8	9396	2012	2085	2012
q9	6631	6491	6508	6491
q10	7069	2443	2452	2443
q11	463	267	262	262
q12	404	219	208	208
q13	17779	3063	3009	3009
q14	243	215	212	212
q15	569	532	505	505
q16	662	611	577	577
q17	983	632	513	513
q18	7415	6708	6636	6636
q19	1329	985	1019	985
q20	473	184	188	184
q21	3988	3225	3221	3221
q22	1108	1038	989	989
Total cold run time: 108546 ms
Total hot run time: 41542 ms
----- Round 2, with runtime_filter_mode=off -----
q1	7301	7302	7281	7281
q2	342	258	255	255
q3	3098	2965	2983	2965
q4	2119	1826	1835	1826
q5	5759	5788	5886	5788
q6	227	148	144	144
q7	2234	1812	1800	1800
q8	3507	3564	3508	3508
q9	9020	8940	8905	8905
q10	3590	3588	3555	3555
q11	612	506	519	506
q12	813	629	589	589
q13	9269	3238	3262	3238
q14	301	270	289	270
q15	610	566	559	559
q16	683	657	667	657
q17	1844	1655	1601	1601
q18	8440	7905	7679	7679
q19	1747	1686	1644	1644
q20	2131	1869	1993	1869
q21	5626	5635	5535	5535
q22	1172	1056	1045	1045
Total cold run time: 70445 ms
Total hot run time: 61219 ms

doris-robot · 2024-11-06T08:16:28Z

TPC-DS: Total hot run time: 196012 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 35b58b30654b8f4a0b87668407395a983c99a68c, data reload: false
query1	2272	2088	2077	2077
query2	6244	2063	2097	2063
query3	15731	11612	238	238
query4	33432	23626	23744	23626
query5	3421	468	453	453
query6	273	194	187	187
query7	3985	297	297	297
query8	290	237	234	234
query9	9287	2728	2724	2724
query10	451	263	248	248
query11	17981	15409	15313	15313
query12	153	103	101	101
query13	1611	433	417	417
query14	8584	7862	7855	7855
query15	248	180	186	180
query16	7368	486	496	486
query17	1599	600	625	600
query18	1630	620	635	620
query19	394	188	186	186
query20	125	125	118	118
query21	209	105	103	103
query22	4748	4537	4408	4408
query23	34799	34223	34228	34223
query24	11265	3343	3389	3343
query25	582	395	402	395
query26	748	175	174	174
query27	1946	279	284	279
query28	6400	2426	2438	2426
query29	680	434	432	432
query30	397	309	309	309
query31	1032	777	811	777
query32	95	56	58	56
query33	710	288	276	276
query34	896	507	515	507
query35	887	764	731	731
query36	1085	958	956	956
query37	120	76	69	69
query38	4501	4331	4469	4331
query39	1490	1407	1449	1407
query40	204	99	100	99
query41	49	47	48	47
query42	109	95	99	95
query43	552	502	501	501
query44	1196	828	815	815
query45	183	169	167	167
query46	1156	701	677	677
query47	1923	1848	1851	1848
query48	414	321	322	321
query49	902	396	417	396
query50	813	395	407	395
query51	7328	7207	7172	7172
query52	102	90	86	86
query53	249	176	172	172
query54	1281	403	410	403
query55	79	73	76	73
query56	255	227	246	227
query57	1238	1187	1131	1131
query58	224	205	196	196
query59	3316	2982	2915	2915
query60	275	241	243	241
query61	111	111	108	108
query62	875	669	765	669
query63	215	193	185	185
query64	3952	659	645	645
query65	3300	3206	3236	3206
query66	835	300	304	300
query67	16016	15604	15580	15580
query68	4458	583	578	578
query69	423	255	259	255
query70	1248	1168	1056	1056
query71	332	255	255	255
query72	6293	4100	3918	3918
query73	756	356	359	356
query74	10266	9025	8986	8986
query75	3403	2683	2747	2683
query76	2903	1143	1168	1143
query77	373	272	269	269
query78	10225	9385	9394	9385
query79	1769	602	590	590
query80	1232	426	429	426
query81	570	242	244	242
query82	990	116	115	115
query83	247	161	158	158
query84	235	81	71	71
query85	1333	320	302	302
query86	426	307	308	307
query87	4909	4681	4659	4659
query88	3354	2199	2160	2160
query89	418	294	292	292
query90	1951	186	187	186
query91	139	105	105	105
query92	59	50	50	50
query93	2142	538	539	538
query94	928	295	300	295
query95	351	247	243	243
query96	611	275	282	275
query97	2911	2710	2718	2710
query98	218	203	203	203
query99	1597	1340	1306	1306
Total cold run time: 301891 ms
Total hot run time: 196012 ms

doris-robot · 2024-11-06T08:21:57Z

ClickBench: Total hot run time: 32.55 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 35b58b30654b8f4a0b87668407395a983c99a68c, data reload: false
query1	0.03	0.03	0.03
query2	0.07	0.03	0.03
query3	0.24	0.06	0.06
query4	1.64	0.10	0.11
query5	0.40	0.39	0.42
query6	1.18	0.65	0.65
query7	0.02	0.02	0.02
query8	0.04	0.04	0.03
query9	0.56	0.51	0.48
query10	0.57	0.56	0.55
query11	0.15	0.11	0.10
query12	0.14	0.10	0.11
query13	0.61	0.60	0.60
query14	2.71	2.75	2.81
query15	0.91	0.84	0.83
query16	0.40	0.38	0.39
query17	0.98	0.99	1.07
query18	0.20	0.19	0.20
query19	1.84	1.87	1.87
query20	0.01	0.01	0.03
query21	15.37	0.63	0.62
query22	2.84	2.09	1.80
query23	16.89	1.05	0.95
query24	2.63	1.00	0.71
query25	0.22	0.16	0.11
query26	0.37	0.15	0.14
query27	0.06	0.05	0.04
query28	11.34	1.10	1.08
query29	12.54	3.25	3.21
query30	0.25	0.06	0.06
query31	2.87	0.39	0.38
query32	3.28	0.47	0.46
query33	3.01	3.00	3.01
query34	17.14	4.57	4.50
query35	4.51	4.57	4.53
query36	0.65	0.48	0.49
query37	0.08	0.06	0.06
query38	0.04	0.04	0.04
query39	0.03	0.02	0.02
query40	0.16	0.13	0.13
query41	0.08	0.02	0.02
query42	0.04	0.02	0.02
query43	0.04	0.04	0.03
Total cold run time: 107.14 s
Total hot run time: 32.55 s

dataroaring

LGTM

sync delete bitmap may cost too much time,which may lead to full compaction fail, but full compaction no need to sync it. pick pr:#43337 Co-authored-by: huanghaibin <huanghaibin@selectdb.com>

…#43337) sync delete bitmap may cost too much time,which may lead to full compaction fail, but full compaction no need to sync it.

[fix](cloud-mow)Full compaction no need to sync delete bitmap
b5d0a2c

edit
35b58b3

github-actionsBot reviewed Nov 6, 2024
View reviewed changes

gavinchou approved these changes Nov 6, 2024
View reviewed changes

github-actionsBot added approved Indicates a PR has been approved by one committer. reviewed labels Nov 6, 2024

dataroaring approved these changes Nov 8, 2024
View reviewed changes

dataroaring merged commit 1216490 into apache:masterNov 8, 2024

dataroaring added the dev/3.0.x label Nov 8, 2024

hust-hhb mentioned this pull request Nov 11, 2024
[fix](cloud-mow)Full compaction no need to sync delete bitmap #43613
Merged

dataroaring added dev/3.0.3-merged and removed dev/3.0.x labels Jun 24, 2025

asdf2014 mentioned this pull request Jul 28, 2026
Cloud MOW: query-visible tablet cache can hold an incomplete delete bitmap after a full-compaction load, and later syncs never repair it #66192
Open
2 tasks

Navigation Menu

Uh oh!

[fix](cloud-mow)Full compaction no need to sync delete bitmap - #43337

[fix](cloud-mow)Full compaction no need to sync delete bitmap#43337
dataroaring merged 2 commits into
apache:masterfrom
hust-hhb:no-nedd-sync

hust-hhb commented Nov 6, 2024

Uh oh!

doris-robot commented Nov 6, 2024

Uh oh!

hust-hhb commented Nov 6, 2024

Uh oh!

github-actionsBot left a comment

Uh oh!

github-actionsBotNov 6, 2024

Uh oh!

hust-hhb commented Nov 6, 2024

Uh oh!

github-actionsBot commented Nov 6, 2024

Uh oh!

github-actionsBot commented Nov 6, 2024

Uh oh!

doris-robot commented Nov 6, 2024

Uh oh!

doris-robot commented Nov 6, 2024

Uh oh!

doris-robot commented Nov 6, 2024

Uh oh!

doris-robot commented Nov 6, 2024

Uh oh!

dataroaring left a comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Uh oh!

Conversation

hust-hhb commented Nov 6, 2024

Uh oh!

doris-robot commented Nov 6, 2024

Uh oh!

hust-hhb commented Nov 6, 2024

Uh oh!

github-actionsBot left a comment

Choose a reason for hiding this comment

Uh oh!

github-actionsBotNov 6, 2024

Choose a reason for hiding this comment

Uh oh!

hust-hhb commented Nov 6, 2024

Uh oh!

github-actionsBot commented Nov 6, 2024

Uh oh!

github-actionsBot commented Nov 6, 2024

Uh oh!

doris-robot commented Nov 6, 2024

Uh oh!

doris-robot commented Nov 6, 2024

Uh oh!

doris-robot commented Nov 6, 2024

Uh oh!

doris-robot commented Nov 6, 2024

Uh oh!

dataroaring left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants