[fix](cloud-mow) MS should delete the existing keys before rewriting it when processing old version delete bitmap on cu compaction by hust-hhb · Pull Request #42379 · apache/doris

hust-hhb · 2024-10-24T04:37:45Z

pr #40204 support deleting old version delete bitmap when doing cu compaction, it will update delete bitmap with agg result, then deleting old version delete bitmap. Updating delete bitmap means overwriting existing keys, however delete bitmap may split into multiple kvs to store in fdb, so we should delete the existing keys and rewrite it to fdb instead of overwriting directly.

doris-robot · 2024-10-24T04:37:49Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

hust-hhb · 2024-10-24T04:38:48Z

run buildall

github-actions · 2024-10-24T04:42:52Z

clang-tidy review says "All clean, LGTM! 👍"

doris-robot · 2024-10-24T05:00:10Z

TPC-H: Total hot run time: 41147 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 365cfa75fff1def6a079f8d88a0de5b54a2e2178, data reload: false
------ Round 1 ----------------------------------
q1	17579	7645	7337	7337
q2	2050	156	170	156
q3	10651	1144	1211	1144
q4	10278	814	776	776
q5	7754	3075	3094	3075
q6	245	157	155	155
q7	1017	611	595	595
q8	9350	1941	1943	1941
q9	6617	6440	6395	6395
q10	7046	2400	2436	2400
q11	444	245	247	245
q12	410	226	231	226
q13	17795	2988	3019	2988
q14	246	213	217	213
q15	573	522	523	522
q16	636	589	578	578
q17	987	539	557	539
q18	7365	6582	6713	6582
q19	1370	990	917	917
q20	479	191	183	183
q21	4312	3171	3192	3171
q22	1090	1009	1010	1009
Total cold run time: 108294 ms
Total hot run time: 41147 ms
----- Round 2, with runtime_filter_mode=off -----
q1	7304	7263	7236	7236
q2	330	234	238	234
q3	2916	2813	2763	2763
q4	1978	1794	1793	1793
q5	5463	5504	5556	5504
q6	231	145	147	145
q7	2145	1724	1748	1724
q8	3265	3410	3414	3410
q9	8571	8595	8567	8567
q10	3539	3470	3425	3425
q11	588	481	475	475
q12	803	584	582	582
q13	11571	3043	3052	3043
q14	301	286	263	263
q15	565	521	524	521
q16	664	633	627	627
q17	1812	1605	1579	1579
q18	7835	7384	7308	7308
q19	1675	1557	1501	1501
q20	2092	1780	1828	1780
q21	5415	5315	5299	5299
q22	1130	1026	1018	1018
Total cold run time: 70193 ms
Total hot run time: 58797 ms

doris-robot · 2024-10-24T05:06:13Z

TeamCity be ut coverage result:
Function Coverage: 37.47% (9709/25913)
Line Coverage: 28.74% (80543/280264)
Region Coverage: 28.16% (41633/147838)
Branch Coverage: 24.73% (21153/85542)
Coverage Report: http://coverage.selectdb-in.cc/coverage/365cfa75fff1def6a079f8d88a0de5b54a2e2178_365cfa75fff1def6a079f8d88a0de5b54a2e2178/report/index.html

doris-robot · 2024-10-24T05:12:18Z

TPC-DS: Total hot run time: 192399 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 365cfa75fff1def6a079f8d88a0de5b54a2e2178, data reload: false
query1	959	385	372	372
query2	6522	2104	2152	2104
query3	6781	220	231	220
query4	33569	23666	23578	23578
query5	4347	478	481	478
query6	262	175	163	163
query7	4613	292	288	288
query8	301	241	242	241
query9	9770	2640	2642	2640
query10	453	268	263	263
query11	18270	15235	15136	15136
query12	153	101	98	98
query13	1658	425	396	396
query14	10123	7614	6993	6993
query15	262	176	178	176
query16	8120	494	504	494
query17	1678	576	555	555
query18	2146	299	313	299
query19	372	154	152	152
query20	119	110	109	109
query21	215	104	105	104
query22	4508	4417	4311	4311
query23	34758	34001	34159	34001
query24	11119	2816	2761	2761
query25	681	409	415	409
query26	1396	160	163	160
query27	2820	282	291	282
query28	7638	2423	2395	2395
query29	909	428	420	420
query30	330	163	161	161
query31	1027	812	816	812
query32	101	60	61	60
query33	804	311	290	290
query34	940	509	529	509
query35	1048	895	899	895
query36	1126	956	974	956
query37	155	86	91	86
query38	4476	4350	4311	4311
query39	1475	1453	1421	1421
query40	281	104	102	102
query41	52	46	45	45
query42	123	102	96	96
query43	555	504	518	504
query44	1238	801	810	801
query45	197	167	164	164
query46	1140	697	682	682
query47	1976	1851	1882	1851
query48	424	316	322	316
query49	1156	436	435	435
query50	833	393	390	390
query51	7197	7085	7031	7031
query52	102	90	89	89
query53	260	184	179	179
query54	1325	441	450	441
query55	80	79	76	76
query56	298	297	259	259
query57	1330	1142	1180	1142
query58	249	251	263	251
query59	3310	3135	3224	3135
query60	299	281	285	281
query61	133	143	104	104
query62	861	677	667	667
query63	222	187	180	180
query64	5333	641	620	620
query65	3388	3262	3209	3209
query66	1450	321	318	318
query67	16159	15622	15769	15622
query68	4919	571	542	542
query69	458	298	313	298
query70	1260	1117	1150	1117
query71	351	273	271	271
query72	6387	4015	3994	3994
query73	773	356	352	352
query74	10214	9164	8988	8988
query75	3435	2708	2692	2692
query76	2886	974	986	974
query77	465	301	288	288
query78	10647	9606	9625	9606
query79	1091	579	601	579
query80	1467	475	449	449
query81	577	245	247	245
query82	1334	138	144	138
query83	236	140	137	137
query84	262	71	75	71
query85	1069	290	283	283
query86	322	315	289	289
query87	4888	4873	4623	4623
query88	3495	2228	2178	2178
query89	405	291	285	285
query90	1997	188	190	188
query91	152	101	104	101
query92	69	48	53	48
query93	1068	552	538	538
query94	773	307	296	296
query95	350	255	252	252
query96	610	278	288	278
query97	2843	2725	2717	2717
query98	212	222	194	194
query99	1747	1299	1308	1299
Total cold run time: 302737 ms
Total hot run time: 192399 ms

doris-robot · 2024-10-24T05:17:47Z

ClickBench: Total hot run time: 32.27 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 365cfa75fff1def6a079f8d88a0de5b54a2e2178, data reload: false
query1	0.04	0.03	0.03
query2	0.06	0.03	0.03
query3	0.23	0.06	0.07
query4	1.65	0.11	0.10
query5	0.42	0.40	0.39
query6	1.17	0.66	0.66
query7	0.02	0.02	0.01
query8	0.06	0.03	0.03
query9	0.58	0.51	0.50
query10	0.56	0.54	0.55
query11	0.14	0.10	0.10
query12	0.13	0.11	0.11
query13	0.61	0.60	0.60
query14	2.81	2.76	2.74
query15	0.91	0.83	0.82
query16	0.38	0.38	0.38
query17	1.07	1.02	1.07
query18	0.24	0.23	0.22
query19	1.95	1.86	2.00
query20	0.01	0.02	0.01
query21	15.35	0.59	0.58
query22	2.53	2.61	1.86
query23	16.96	0.91	0.77
query24	3.21	0.54	1.47
query25	0.20	0.12	0.16
query26	0.53	0.13	0.14
query27	0.04	0.04	0.04
query28	10.50	1.10	1.07
query29	12.55	3.32	3.31
query30	0.25	0.07	0.06
query31	2.87	0.38	0.39
query32	3.25	0.47	0.46
query33	2.98	3.00	3.03
query34	16.90	4.49	4.58
query35	4.47	4.50	4.47
query36	0.67	0.50	0.48
query37	0.09	0.06	0.06
query38	0.05	0.03	0.03
query39	0.04	0.02	0.02
query40	0.16	0.14	0.12
query41	0.08	0.02	0.03
query42	0.03	0.02	0.02
query43	0.03	0.03	0.03
Total cold run time: 106.78 s
Total hot run time: 32.27 s

github-actions

clang-tidy made some suggestions

hust-hhb · 2024-10-24T12:55:37Z

run buildall

doris-robot · 2024-10-24T13:14:32Z

TPC-H: Total hot run time: 41566 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 0e9b81114c0bc163e7ab6803e00f86d2e2521277, data reload: false
------ Round 1 ----------------------------------
q1	17586	7408	7321	7321
q2	2043	186	158	158
q3	10552	1158	1121	1121
q4	10234	880	860	860
q5	7741	3066	3054	3054
q6	236	151	149	149
q7	1027	627	619	619
q8	9399	1956	1988	1956
q9	6540	6405	6407	6405
q10	7085	2487	2410	2410
q11	443	250	251	250
q12	422	227	219	219
q13	17818	3052	3053	3052
q14	234	210	219	210
q15	561	515	498	498
q16	635	576	608	576
q17	983	653	573	573
q18	7513	6781	6777	6777
q19	1346	1005	979	979
q20	494	186	185	185
q21	4121	3470	3192	3192
q22	1102	1016	1002	1002
Total cold run time: 108115 ms
Total hot run time: 41566 ms
----- Round 2, with runtime_filter_mode=off -----
q1	7330	7251	7253	7251
q2	328	234	229	229
q3	2980	2809	2863	2809
q4	1987	1722	1753	1722
q5	5515	5461	5516	5461
q6	217	140	139	139
q7	2159	1708	1726	1708
q8	3226	3390	3445	3390
q9	8593	8647	8585	8585
q10	3496	3425	3424	3424
q11	589	509	479	479
q12	818	643	595	595
q13	10001	3068	3049	3049
q14	307	263	277	263
q15	559	522	515	515
q16	677	657	627	627
q17	1801	1620	1590	1590
q18	7804	7510	7405	7405
q19	1660	1487	1507	1487
q20	2043	1825	1812	1812
q21	5524	5362	5324	5324
q22	1087	1018	985	985
Total cold run time: 68701 ms
Total hot run time: 58849 ms

doris-robot · 2024-10-24T13:19:24Z

TeamCity be ut coverage result:
Function Coverage: 37.46% (9712/25923)
Line Coverage: 28.72% (80533/280377)
Region Coverage: 28.15% (41647/147931)
Branch Coverage: 24.73% (21160/85574)
Coverage Report: http://coverage.selectdb-in.cc/coverage/0e9b81114c0bc163e7ab6803e00f86d2e2521277_0e9b81114c0bc163e7ab6803e00f86d2e2521277/report/index.html

doris-robot · 2024-10-24T13:26:41Z

TPC-DS: Total hot run time: 191310 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 0e9b81114c0bc163e7ab6803e00f86d2e2521277, data reload: false
query1	973	370	365	365
query2	6508	2082	2113	2082
query3	6792	221	222	221
query4	34321	23536	23532	23532
query5	4383	461	451	451
query6	268	192	165	165
query7	4606	305	305	305
query8	290	236	228	228
query9	9642	2732	2731	2731
query10	488	256	260	256
query11	18274	15220	15373	15220
query12	157	104	101	101
query13	1687	423	422	422
query14	10544	7234	6818	6818
query15	254	168	180	168
query16	8099	442	458	442
query17	1657	563	571	563
query18	2154	314	299	299
query19	359	162	158	158
query20	116	108	115	108
query21	211	108	112	108
query22	4589	4093	4442	4093
query23	34935	34480	33965	33965
query24	11073	2839	2817	2817
query25	671	397	406	397
query26	1210	162	161	161
query27	2818	284	286	284
query28	8128	2446	2446	2446
query29	869	424	437	424
query30	326	171	161	161
query31	1023	777	814	777
query32	99	58	60	58
query33	786	273	273	273
query34	979	516	547	516
query35	1038	876	888	876
query36	1099	959	938	938
query37	129	75	71	71
query38	4412	4262	4259	4259
query39	1456	1438	1420	1420
query40	203	101	103	101
query41	48	48	47	47
query42	114	105	99	99
query43	533	499	493	493
query44	1252	801	806	801
query45	186	170	168	168
query46	1122	691	696	691
query47	1933	1839	1845	1839
query48	421	340	330	330
query49	1175	419	452	419
query50	811	387	396	387
query51	7122	6975	6972	6972
query52	103	91	89	89
query53	259	184	187	184
query54	1290	428	425	425
query55	80	79	80	79
query56	260	242	238	238
query57	1344	1186	1165	1165
query58	246	214	208	208
query59	3482	3314	2939	2939
query60	280	253	252	252
query61	109	102	102	102
query62	876	675	670	670
query63	213	189	191	189
query64	5272	652	615	615
query65	3291	3235	3250	3235
query66	1439	301	302	301
query67	15845	15692	15613	15613
query68	5003	572	568	568
query69	455	264	262	262
query70	1224	1169	1117	1117
query71	346	267	261	261
query72	6360	4072	3923	3923
query73	763	360	371	360
query74	10476	8999	8889	8889
query75	3366	2641	2676	2641
query76	2932	997	993	993
query77	491	291	306	291
query78	10612	9568	9518	9518
query79	2504	618	626	618
query80	1121	430	422	422
query81	567	242	241	241
query82	809	120	116	116
query83	208	136	147	136
query84	229	74	75	74
query85	1617	322	294	294
query86	477	284	298	284
query87	4881	4611	4634	4611
query88	4086	2192	2176	2176
query89	398	285	288	285
query90	2090	189	187	187
query91	133	101	110	101
query92	61	50	50	50
query93	1588	562	549	549
query94	978	299	294	294
query95	360	255	245	245
query96	615	281	279	279
query97	2861	2724	2683	2683
query98	211	199	198	198
query99	1539	1317	1314	1314
Total cold run time: 306093 ms
Total hot run time: 191310 ms

doris-robot · 2024-10-24T13:32:07Z

ClickBench: Total hot run time: 32.43 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 0e9b81114c0bc163e7ab6803e00f86d2e2521277, data reload: false
query1	0.03	0.03	0.03
query2	0.07	0.03	0.02
query3	0.23	0.06	0.06
query4	1.64	0.10	0.10
query5	0.41	0.38	0.40
query6	1.19	0.66	0.66
query7	0.02	0.01	0.02
query8	0.06	0.03	0.04
query9	0.56	0.49	0.53
query10	0.54	0.54	0.54
query11	0.14	0.11	0.11
query12	0.14	0.11	0.11
query13	0.61	0.60	0.61
query14	2.83	2.81	2.74
query15	0.90	0.83	0.83
query16	0.37	0.39	0.39
query17	1.04	1.05	1.02
query18	0.24	0.23	0.22
query19	1.94	1.89	2.01
query20	0.02	0.01	0.01
query21	15.37	0.60	0.57
query22	2.84	2.16	1.53
query23	17.11	0.86	0.75
query24	2.60	1.13	1.28
query25	0.22	0.11	0.27
query26	0.38	0.13	0.12
query27	0.04	0.04	0.04
query28	10.63	1.11	1.07
query29	12.59	3.33	3.30
query30	0.25	0.06	0.05
query31	2.88	0.39	0.39
query32	3.25	0.46	0.47
query33	2.98	3.05	3.03
query34	16.91	4.44	4.40
query35	4.52	4.51	4.48
query36	0.67	0.47	0.48
query37	0.09	0.06	0.06
query38	0.05	0.04	0.03
query39	0.03	0.02	0.02
query40	0.16	0.13	0.12
query41	0.08	0.02	0.03
query42	0.03	0.03	0.02
query43	0.03	0.02	0.02
Total cold run time: 106.69 s
Total hot run time: 32.43 s

github-actions

clang-tidy made some suggestions

github-actions · 2024-10-28T04:14:53Z

 << " total_size=" << total_size << " unlock=" << unlock;
 }

+void MetaServiceImpl::update_delete_bitmap2(google::protobuf::RpcController* controller,


warning: function 'update_delete_bitmap2' exceeds recommended size/complexity thresholds [readability-function-size]
voidMetaServiceImpl::update_delete_bitmap2(google::protobuf::RpcController* controller, ^
Additional context
cloud/src/meta-service/meta_service.cpp:1858: 132 lines including whitespace and comments (threshold 80)
voidMetaServiceImpl::update_delete_bitmap2(google::protobuf::RpcController* controller, ^

github-actions

clang-tidy made some suggestions

hust-hhb · 2024-10-31T01:26:59Z

run buildall

github-actions · 2024-10-31T01:31:08Z

clang-tidy review says "All clean, LGTM! 👍"

hust-hhb · 2024-10-31T01:50:32Z

run buildall

hust-hhb · 2024-10-31T01:52:03Z

run buildall

github-actions · 2024-10-31T01:54:56Z

clang-tidy review says "All clean, LGTM! 👍"

github-actions · 2024-10-31T01:56:54Z

clang-tidy review says "All clean, LGTM! 👍"

doris-robot · 2024-10-31T03:44:52Z

TeamCity be ut coverage result:
Function Coverage: 37.91% (9847/25976)
Line Coverage: 29.16% (81967/281111)
Region Coverage: 28.44% (42316/148812)
Branch Coverage: 25.03% (21499/85908)
Coverage Report: http://coverage.selectdb-in.cc/coverage/0e18e47b8fc76d42d5edad8b75450a1d1d52917c_0e18e47b8fc76d42d5edad8b75450a1d1d52917c/report/index.html

…itmap when doing cu compaction

github-actions · 2024-10-31T06:39:14Z

clang-tidy review says "All clean, LGTM! 👍"

doris-robot · 2024-11-01T07:08:12Z

TPC-DS: Total hot run time: 197711 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 3564f1ea993ab66ca07c3c005eb32e6ef058412d, data reload: false
query1	1223	923	916	916
query2	6206	2124	2111	2111
query3	10765	3986	4136	3986
query4	67452	29655	23701	23701
query5	4964	462	432	432
query6	403	173	163	163
query7	5637	302	312	302
query8	324	228	224	224
query9	9301	2740	2741	2740
query10	453	255	246	246
query11	17672	15579	15784	15579
query12	163	100	109	100
query13	1609	448	429	429
query14	10049	7820	7224	7224
query15	208	173	174	173
query16	7047	464	463	463
query17	1020	566	579	566
query18	1783	305	295	295
query19	211	157	150	150
query20	116	113	111	111
query21	202	97	102	97
query22	4621	4353	4419	4353
query23	34546	34085	34423	34085
query24	6010	2832	2783	2783
query25	515	417	412	412
query26	660	168	159	159
query27	1681	287	294	287
query28	4128	2504	2461	2461
query29	689	429	432	429
query30	226	156	154	154
query31	997	813	799	799
query32	70	58	62	58
query33	458	275	264	264
query34	902	523	533	523
query35	866	737	742	737
query36	1067	940	931	931
query37	128	74	71	71
query38	4422	4326	4373	4326
query39	1480	1436	1440	1436
query40	200	101	101	101
query41	49	45	49	45
query42	110	101	103	101
query43	545	505	509	505
query44	1192	819	835	819
query45	189	174	168	168
query46	1118	718	700	700
query47	1958	1865	1840	1840
query48	431	349	322	322
query49	737	423	408	408
query50	812	398	395	395
query51	7349	7045	7148	7045
query52	102	87	91	87
query53	258	185	200	185
query54	544	417	407	407
query55	76	75	79	75
query56	261	241	257	241
query57	1275	1148	1155	1148
query58	214	204	225	204
query59	3225	3217	3212	3212
query60	261	260	241	241
query61	109	101	104	101
query62	787	665	659	659
query63	215	189	190	189
query64	1357	619	614	614
query65	3298	3242	3246	3242
query66	752	309	310	309
query67	16019	15938	15881	15881
query68	3238	612	580	580
query69	413	258	250	250
query70	1180	1081	1029	1029
query71	369	252	251	251
query72	6139	4032	3975	3975
query73	773	373	367	367
query74	10089	9119	9118	9118
query75	3414	2670	2693	2670
query76	1757	1071	1104	1071
query77	477	281	283	281
query78	10421	9485	9389	9389
query79	1369	607	596	596
query80	849	429	445	429
query81	514	239	240	239
query82	1176	119	119	119
query83	164	146	140	140
query84	281	72	72	72
query85	809	302	284	284
query86	326	286	304	286
query87	4758	4772	4662	4662
query88	3248	2245	2203	2203
query89	416	298	287	287
query90	2026	191	188	188
query91	134	102	99	99
query92	65	49	46	46
query93	1557	553	568	553
query94	771	300	285	285
query95	342	247	263	247
query96	620	280	292	280
query97	2900	2734	2687	2687
query98	226	211	203	203
query99	1552	1307	1305	1305
Total cold run time: 316090 ms
Total hot run time: 197711 ms

doris-robot · 2024-11-01T07:13:42Z

ClickBench: Total hot run time: 32.38 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 3564f1ea993ab66ca07c3c005eb32e6ef058412d, data reload: false
query1	0.04	0.03	0.04
query2	0.07	0.03	0.04
query3	0.23	0.07	0.07
query4	1.63	0.11	0.11
query5	0.42	0.39	0.43
query6	1.16	0.65	0.66
query7	0.02	0.02	0.01
query8	0.04	0.04	0.03
query9	0.56	0.48	0.50
query10	0.55	0.55	0.57
query11	0.15	0.11	0.11
query12	0.14	0.10	0.11
query13	0.60	0.59	0.60
query14	2.81	2.86	2.76
query15	0.91	0.83	0.83
query16	0.38	0.38	0.39
query17	1.03	1.07	0.99
query18	0.19	0.19	0.20
query19	1.94	1.78	1.95
query20	0.02	0.01	0.01
query21	15.39	0.58	0.59
query22	2.82	1.95	2.09
query23	17.06	0.78	0.90
query24	2.60	1.05	0.87
query25	0.22	0.18	0.05
query26	0.44	0.14	0.14
query27	0.04	0.04	0.04
query28	11.08	1.10	1.08
query29	12.59	3.25	3.24
query30	0.24	0.06	0.05
query31	2.88	0.37	0.38
query32	3.26	0.46	0.46
query33	2.96	3.08	3.00
query34	17.13	4.43	4.47
query35	4.49	4.51	4.49
query36	0.67	0.49	0.47
query37	0.08	0.05	0.06
query38	0.06	0.03	0.03
query39	0.04	0.02	0.02
query40	0.16	0.12	0.13
query41	0.08	0.02	0.03
query42	0.03	0.02	0.02
query43	0.04	0.03	0.03
Total cold run time: 107.25 s
Total hot run time: 32.38 s

zhannngchen

LGTM

github-actions · 2024-11-01T10:00:34Z

PR approved by at least one committer and no changes requested.

github-actions · 2024-11-01T10:00:37Z

PR approved by anyone and no changes requested.

hust-hhb · 2024-11-01T12:58:16Z

run buildall

github-actions · 2024-11-01T13:02:48Z

clang-tidy review says "All clean, LGTM! 👍"

doris-robot · 2024-11-01T14:20:42Z

TPC-H: Total hot run time: 41684 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 74f1a52095d6f08bfa92c796446509805e467fa0, data reload: false
------ Round 1 ----------------------------------
q1	17578	7470	7351	7351
q2	2056	168	167	167
q3	10556	1097	1216	1097
q4	10229	870	837	837
q5	7758	3144	3123	3123
q6	235	145	147	145
q7	1018	603	609	603
q8	9368	2052	2041	2041
q9	6702	6493	6501	6493
q10	7049	2464	2425	2425
q11	460	264	266	264
q12	410	234	223	223
q13	17821	3075	3042	3042
q14	244	216	207	207
q15	586	537	529	529
q16	661	599	575	575
q17	996	556	520	520
q18	7537	6716	6729	6716
q19	1332	1034	971	971
q20	480	184	190	184
q21	4223	3276	3176	3176
q22	1133	997	995	995
Total cold run time: 108432 ms
Total hot run time: 41684 ms
----- Round 2, with runtime_filter_mode=off -----
q1	7300	7289	7305	7289
q2	335	227	227	227
q3	3107	2980	2972	2972
q4	2183	1859	1847	1847
q5	5829	5824	5863	5824
q6	230	144	146	144
q7	2307	1840	1816	1816
q8	3413	3533	3576	3533
q9	8992	8973	8965	8965
q10	3651	3566	3660	3566
q11	597	525	521	521
q12	838	608	675	608
q13	8602	3246	3188	3188
q14	310	281	299	281
q15	595	528	526	526
q16	693	651	662	651
q17	1861	1649	1631	1631
q18	8302	7877	7489	7489
q19	1719	1556	1544	1544
q20	2172	1857	1888	1857
q21	5770	5595	5608	5595
q22	1184	1073	1063	1063
Total cold run time: 69990 ms
Total hot run time: 61137 ms

doris-robot · 2024-11-01T14:31:55Z

TeamCity be ut coverage result:
Function Coverage: 37.81% (9818/25967)
Line Coverage: 28.97% (81568/281550)
Region Coverage: 28.25% (42128/149152)
Branch Coverage: 24.82% (21374/86102)
Coverage Report: http://coverage.selectdb-in.cc/coverage/74f1a52095d6f08bfa92c796446509805e467fa0_74f1a52095d6f08bfa92c796446509805e467fa0/report/index.html

doris-robot · 2024-11-01T14:33:20Z

TPC-DS: Total hot run time: 196504 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 74f1a52095d6f08bfa92c796446509805e467fa0, data reload: false
query1	1206	920	904	904
query2	6273	2152	2144	2144
query3	10908	4006	3987	3987
query4	68576	28907	23631	23631
query5	4881	452	462	452
query6	399	177	166	166
query7	5608	298	291	291
query8	306	229	228	228
query9	9345	2742	2733	2733
query10	445	257	240	240
query11	17603	15386	15955	15386
query12	154	106	110	106
query13	1550	441	433	433
query14	10555	6847	7078	6847
query15	196	165	175	165
query16	7037	481	462	462
query17	1005	560	557	557
query18	1802	292	297	292
query19	192	155	149	149
query20	118	113	109	109
query21	206	97	101	97
query22	4707	4283	4200	4200
query23	34716	34134	34766	34134
query24	6004	2748	2721	2721
query25	491	398	390	390
query26	643	160	157	157
query27	1682	287	290	287
query28	4416	2481	2421	2421
query29	672	424	422	422
query30	240	159	156	156
query31	1019	759	825	759
query32	79	56	60	56
query33	456	271	278	271
query34	892	506	517	506
query35	871	743	740	740
query36	1091	938	960	938
query37	116	74	75	74
query38	4361	4344	4333	4333
query39	1465	1432	1429	1429
query40	204	98	98	98
query41	48	45	46	45
query42	105	97	95	95
query43	529	507	498	498
query44	1160	818	814	814
query45	182	165	167	165
query46	1113	698	758	698
query47	1984	1882	1883	1882
query48	430	326	322	322
query49	740	415	412	412
query50	814	391	391	391
query51	7303	7131	7167	7131
query52	103	93	86	86
query53	253	182	206	182
query54	516	400	387	387
query55	73	76	74	74
query56	243	237	245	237
query57	1269	1204	1133	1133
query58	222	202	202	202
query59	3215	3258	3235	3235
query60	259	240	249	240
query61	100	93	110	93
query62	789	685	670	670
query63	220	185	195	185
query64	1336	612	577	577
query65	3258	3245	3234	3234
query66	707	306	300	300
query67	16099	15779	15838	15779
query68	3480	561	566	561
query69	422	257	248	248
query70	1233	1170	1157	1157
query71	346	260	249	249
query72	5839	3975	4026	3975
query73	754	365	352	352
query74	10005	8988	9092	8988
query75	3388	2648	2648	2648
query76	1775	1189	1092	1092
query77	481	280	279	279
query78	10521	9438	9388	9388
query79	1382	568	587	568
query80	854	408	416	408
query81	514	240	255	240
query82	1285	114	114	114
query83	181	140	136	136
query84	282	77	72	72
query85	864	302	281	281
query86	339	313	313	313
query87	5053	4683	4622	4622
query88	3394	2195	2166	2166
query89	420	292	284	284
query90	2023	189	183	183
query91	131	101	99	99
query92	63	49	49	49
query93	1666	537	532	532
query94	820	289	284	284
query95	351	257	242	242
query96	620	276	288	276
query97	2899	2666	2707	2666
query98	220	202	195	195
query99	1763	1342	1299	1299
Total cold run time: 318928 ms
Total hot run time: 196504 ms

doris-robot · 2024-11-01T14:38:49Z

ClickBench: Total hot run time: 32.66 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 74f1a52095d6f08bfa92c796446509805e467fa0, data reload: false
query1	0.03	0.03	0.03
query2	0.06	0.03	0.03
query3	0.23	0.06	0.06
query4	1.64	0.10	0.09
query5	0.43	0.41	0.39
query6	1.16	0.65	0.66
query7	0.02	0.02	0.01
query8	0.04	0.03	0.03
query9	0.57	0.49	0.49
query10	0.56	0.55	0.55
query11	0.14	0.10	0.12
query12	0.14	0.11	0.11
query13	0.61	0.60	0.60
query14	2.71	2.82	2.71
query15	0.92	0.83	0.82
query16	0.38	0.37	0.38
query17	1.04	1.02	1.07
query18	0.20	0.19	0.20
query19	1.99	1.84	1.96
query20	0.02	0.01	0.01
query21	15.35	0.59	0.59
query22	2.51	1.87	2.08
query23	16.95	1.12	0.95
query24	2.82	1.56	0.99
query25	0.26	0.04	0.06
query26	0.53	0.14	0.14
query27	0.04	0.04	0.04
query28	10.54	1.10	1.06
query29	12.54	3.22	3.20
query30	0.24	0.06	0.06
query31	2.87	0.38	0.38
query32	3.29	0.46	0.46
query33	2.95	3.01	2.97
query34	16.89	4.49	4.52
query35	4.57	4.55	4.57
query36	0.66	0.47	0.49
query37	0.09	0.06	0.06
query38	0.05	0.04	0.03
query39	0.03	0.03	0.02
query40	0.16	0.12	0.12
query41	0.07	0.02	0.02
query42	0.04	0.02	0.02
query43	0.03	0.03	0.03
Total cold run time: 106.37 s
Total hot run time: 32.66 s

github-actions · 2024-11-06T02:03:51Z

PR approved by at least one committer and no changes requested.

…it when processing old version delete bitmap on cu compaction (#43479) pick pr #42379

…it when processing old version delete bitmap on cu compaction (apache#42379) pr apache#40204 support deleting old version delete bitmap when doing cu compaction, it will update delete bitmap with agg result, then deleting old version delete bitmap. Updating delete bitmap means overwriting existing keys, however delete bitmap may split into multiple kvs to store in fdb, so we should delete the existing keys and rewrite it to fdb instead of overwriting directly.

github-actionsBot reviewed Oct 24, 2024
View reviewed changes

Comment threadbe/src/cloud/cloud_cumulative_compaction.cpp

hust-hhb changed the title ~~[fix](cloud-mow) Add enable config for procesing old version delete bitmap when doing cu compaction~~[fix](cloud-mow) Should delete the existing keys before rewriting it when processing old version delete bitmap on cu compactionOct 24, 2024

zhannngchen reviewed Oct 25, 2024
View reviewed changes

Comment threadbe/src/cloud/config.cpp Outdated
Comment threadbe/src/cloud/cloud_cumulative_compaction.cpp Outdated

github-actionsBot reviewed Oct 28, 2024
View reviewed changes

hust-hhb changed the title ~~[fix](cloud-mow) Should delete the existing keys before rewriting it when processing old version delete bitmap on cu compaction~~[fix](cloud-mow) MS should delete the existing keys before rewriting it when processing old version delete bitmap on cu compactionOct 30, 2024

github-actionsBot reviewed Oct 30, 2024
View reviewed changes

Comment threadbe/src/cloud/cloud_cumulative_compaction.cpp

[fix](cloud-mow) Add enable config for procesing old version delete b…
de8d554
…itmap when doing cu compaction

hust-hhb added 3 commits October 31, 2024 14:39

edit
4111dc0

edit
2c1b351

edit11
bddb7c0

zhannngchen previously approved these changes Nov 1, 2024
View reviewed changes

github-actionsBot added the approved Indicates a PR has been approved by one committer. label Nov 1, 2024

github-actionsBot added the reviewed label Nov 1, 2024

dataroaring added dev/3.0.x p0_c labels Nov 1, 2024

edit
74f1a52

hust-hhb dismissed zhannngchen’s stale review via 74f1a52November 1, 2024 12:57

github-actionsBot removed the approved Indicates a PR has been approved by one committer. label Nov 1, 2024

zhannngchen approved these changes Nov 6, 2024
View reviewed changes

github-actionsBot added the approved Indicates a PR has been approved by one committer. label Nov 6, 2024

gavinchou approved these changes Nov 6, 2024
View reviewed changes

dataroaring merged commit eba34e3 into apache:masterNov 7, 2024

github-actionsBot added the cherry-pick-conflict-in-3.0 label Nov 7, 2024

hust-hhb mentioned this pull request Nov 8, 2024
[fix](cloud-mow) MS should delete the existing keys before rewriting it when processing old version delete bitmap on cu compaction #43479
Merged

zhannngchen pushed a commit that referenced this pull request Nov 8, 2024
[fix](cloud-mow) MS should delete the existing keys before rewriting …
3bd553d
…it when processing old version delete bitmap on cu compaction (#43479) pick pr #42379

gavinchou added dev/3.0.3-merged and removed dev/3.0.x labels Nov 14, 2024

dataroaring removed the cherry-pick-conflict-in-3.0 label Nov 16, 2024

Uh oh!

Conversation

hust-hhb commented Oct 24, 2024• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

doris-robot commented Oct 24, 2024

Uh oh!

hust-hhb commented Oct 24, 2024

Uh oh!

github-actionsBot commented Oct 24, 2024

Uh oh!

doris-robot commented Oct 24, 2024

Uh oh!

doris-robot commented Oct 24, 2024

Uh oh!

doris-robot commented Oct 24, 2024

Uh oh!

doris-robot commented Oct 24, 2024

Uh oh!

github-actionsBot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

hust-hhb commented Oct 24, 2024

Uh oh!

doris-robot commented Oct 24, 2024

Uh oh!

doris-robot commented Oct 24, 2024

Uh oh!

doris-robot commented Oct 24, 2024

Uh oh!

doris-robot commented Oct 24, 2024

Uh oh!

Uh oh!

Uh oh!

github-actionsBot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

github-actionsBotOct 28, 2024

Choose a reason for hiding this comment

Uh oh!

github-actionsBot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

hust-hhb commented Oct 31, 2024

Uh oh!

github-actionsBot commented Oct 31, 2024

Uh oh!

hust-hhb commented Oct 31, 2024

Uh oh!

hust-hhb commented Oct 31, 2024

Uh oh!

github-actionsBot commented Oct 31, 2024

Uh oh!

github-actionsBot commented Oct 31, 2024

Uh oh!

doris-robot commented Oct 31, 2024

Uh oh!

github-actionsBot commented Oct 31, 2024

Uh oh!

doris-robot commented Nov 1, 2024

Uh oh!

doris-robot commented Nov 1, 2024

Uh oh!

zhannngchen left a comment

Choose a reason for hiding this comment

Uh oh!

github-actionsBot commented Nov 1, 2024

Uh oh!

github-actionsBot commented Nov 1, 2024

Uh oh!

hust-hhb commented Nov 1, 2024

Uh oh!

github-actionsBot commented Nov 1, 2024

Uh oh!

doris-robot commented Nov 1, 2024

Uh oh!

doris-robot commented Nov 1, 2024

hust-hhb commented Oct 24, 2024•
edited
Loading