[opt](cloud) optimize load performance for mow table when pack small files by liaoxin01 · Pull Request #58839 · apache/doris

liaoxin01 · 2025-12-09T03:05:22Z

What problem does this PR solve?

Issue Number: close #xxx

Related PR: #57770

Problem Summary:
The main change is to offload the entire delete bitmap calculation process—including file closing, rowset building, segment loading, and bitmap calculation—to a background thread pool. This prevents blocking the memtable flush thread, enhancing performance and concurrency.

Release note

None

Check List (For Author)

Test
- Regression test
- Unit Test
- Manual test (add detailed scripts or steps below)
- No need to test or manual test. Explain why:
  - This is a refactor/code format and no logic has been changed.
  - Previous test can cover this change.
  - No code files have been changed.
  - Other reason
Behavior changed:
- No.
- Yes.
Does this need documentation?
- No.
- Yes.

Check List (For Reviewer who merge this PR)

Confirm the release note
Confirm test cases
Confirm document
Add branch pick label

…files

hello-stephen · 2025-12-09T03:05:28Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

What problem was fixed (it's best to include specific error reporting information). How it was fixed.
Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
What features were added. Why was this function added?
Which code was refactored and why was this part of the code refactored?
Which functions were optimized and what is the difference before and after the optimization?

liaoxin01 · 2025-12-09T03:05:33Z

run buildall

Copilot

Copilot encountered an error and was unable to review this pull request. You can try again by re-requesting a review.

doris-robot · 2025-12-09T04:04:28Z

TPC-H: Total hot run time: 35920 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 993ad294f73d2715022bfa68bdde0e429ae1bf2b, data reload: false
------ Round 1 ----------------------------------
q1	17730	5064	4948	4948
q2	2063	363	252	252
q3	10149	1310	755	755
q4	10229	951	326	326
q5	7547	2143	1946	1946
q6	193	173	143	143
q7	994	850	753	753
q8	9373	1490	1185	1185
q9	6993	5345	5310	5310
q10	6833	2395	1990	1990
q11	564	312	281	281
q12	667	754	600	600
q13	17789	3683	3000	3000
q14	297	297	272	272
q15	609	514	507	507
q16	908	905	878	878
q17	732	797	534	534
q18	7657	7123	6931	6931
q19	1143	957	609	609
q20	400	378	248	248
q21	4316	3871	3509	3509
q22	1049	995	943	943
Total cold run time: 108235 ms
Total hot run time: 35920 ms
----- Round 2, with runtime_filter_mode=off -----
q1	5021	4971	5001	4971
q2	314	396	317	317
q3	2110	2710	2269	2269
q4	1314	1771	1277	1277
q5	4940	4717	4459	4459
q6	215	164	127	127
q7	2019	1965	1796	1796
q8	2622	2517	2495	2495
q9	7601	7513	7495	7495
q10	3048	3387	2842	2842
q11	579	500	487	487
q12	710	766	670	670
q13	3628	3962	3268	3268
q14	282	290	267	267
q15	551	526	520	520
q16	910	941	872	872
q17	1215	1464	1443	1443
q18	7918	7651	7643	7643
q19	900	880	921	880
q20	2028	2030	1809	1809
q21	4760	4261	4083	4083
q22	1095	1048	976	976
Total cold run time: 53780 ms
Total hot run time: 50966 ms

doris-robot · 2025-12-09T04:16:03Z

TPC-DS: Total hot run time: 181221 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 993ad294f73d2715022bfa68bdde0e429ae1bf2b, data reload: false
query5	5131	657	508	508
query6	365	250	228	228
query7	4704	476	284	284
query8	331	268	244	244
query9	8754	2617	2650	2617
query10	610	384	363	363
query11	15220	15128	14687	14687
query12	226	123	116	116
query13	1708	523	423	423
query14	6872	3286	3088	3088
query14_1	2959	2937	2928	2928
query15	214	204	187	187
query16	7699	481	462	462
query17	1313	699	624	624
query18	2031	444	349	349
query19	258	236	207	207
query20	130	115	118	115
query21	285	138	116	116
query22	4012	4051	3913	3913
query23	16969	16225	15960	15960
query23_1	16030	15861	16099	15861
query24	7149	1653	1246	1246
query24_1	1286	1234	1255	1234
query25	644	498	473	473
query26	1347	276	171	171
query27	2809	490	331	331
query28	4366	2230	2152	2152
query29	818	550	441	441
query30	372	242	222	222
query31	820	674	619	619
query32	83	68	70	68
query33	681	343	293	293
query34	895	876	534	534
query35	787	796	732	732
query36	924	959	800	800
query37	130	94	79	79
query38	3949	3840	3810	3810
query39	759	722	723	722
query39_1	705	711	695	695
query40	266	145	126	126
query41	69	64	64	64
query42	131	106	107	106
query43	475	439	411	411
query44	1376	772	770	770
query45	205	191	176	176
query46	968	984	618	618
query47	1660	1733	1630	1630
query48	447	335	250	250
query49	765	438	362	362
query50	689	303	218	218
query51	3800	3842	3819	3819
query52	123	114	98	98
query53	355	363	320	320
query54	342	269	273	269
query55	94	81	78	78
query56	343	303	320	303
query57	1178	1148	1069	1069
query58	287	261	258	258
query59	2340	2451	2290	2290
query60	354	308	317	308
query61	162	158	165	158
query62	864	663	615	615
query63	342	299	304	299
query64	4512	1307	1051	1051
query65	4065	3921	3960	3921
query66	1157	433	327	327
query67	14939	14976	14726	14726
query68	8266	1018	749	749
query69	560	357	315	315
query70	1114	1041	980	980
query71	487	320	302	302
query72	5785	4785	4806	4785
query73	686	563	306	306
query74	8858	8693	8642	8642
query75	3726	3526	3160	3160
query76	3740	1135	768	768
query77	828	402	295	295
query78	9411	9586	8818	8818
query79	1683	879	642	642
query80	733	643	560	560
query81	520	268	237	237
query82	233	131	103	103
query83	257	258	235	235
query84	262	138	106	106
query85	929	523	478	478
query86	387	295	270	270
query87	4052	4080	3872	3872
query88	4456	2343	2308	2308
query89	497	442	393	393
query90	2126	164	158	158
query91	170	168	149	149
query92	83	67	66	66
query93	1757	921	568	568
query94	738	310	278	278
query95	567	369	317	317
query96	583	479	211	211
query97	2600	2688	2580	2580
query98	238	189	194	189
query99	1350	1281	1247	1247
Total cold run time: 275083 ms
Total hot run time: 181221 ms

doris-robot · 2025-12-09T04:21:06Z

ClickBench: Total hot run time: 27.51 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 993ad294f73d2715022bfa68bdde0e429ae1bf2b, data reload: false
query1	0.06	0.05	0.05
query2	0.10	0.04	0.04
query3	0.26	0.08	0.08
query4	1.62	0.11	0.11
query5	0.27	0.26	0.24
query6	1.18	0.64	0.63
query7	0.03	0.02	0.03
query8	0.06	0.04	0.04
query9	0.57	0.52	0.51
query10	0.56	0.57	0.56
query11	0.15	0.10	0.10
query12	0.15	0.12	0.12
query13	0.62	0.59	0.60
query14	1.00	0.98	0.98
query15	0.80	0.80	0.80
query16	0.42	0.39	0.41
query17	1.07	1.03	1.05
query18	0.23	0.21	0.21
query19	1.87	1.86	1.91
query20	0.02	0.01	0.01
query21	15.46	0.27	0.13
query22	4.85	0.06	0.04
query23	16.00	0.28	0.10
query24	1.23	0.61	0.56
query25	0.10	0.08	0.06
query26	0.16	0.13	0.13
query27	0.05	0.08	0.04
query28	4.63	1.20	1.02
query29	12.58	4.09	3.16
query30	0.28	0.13	0.13
query31	2.82	0.63	0.39
query32	3.23	0.56	0.45
query33	3.13	3.01	3.09
query34	16.94	5.18	4.51
query35	4.58	4.56	4.59
query36	0.68	0.51	0.49
query37	0.11	0.07	0.07
query38	0.08	0.05	0.05
query39	0.04	0.03	0.04
query40	0.17	0.13	0.13
query41	0.09	0.03	0.02
query42	0.05	0.03	0.03
query43	0.04	0.04	0.03
Total cold run time: 98.34 s
Total hot run time: 27.51 s

hello-stephen · 2025-12-09T04:51:29Z

BE UT Coverage Report

Increment line coverage 66.67% (62/93) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	53.35% (18713/35074)
Line Coverage	39.07% (173050/442976)
Region Coverage	33.74% (134197/397759)
Branch Coverage	34.63% (57635/166410)

liaoxin01 · 2025-12-09T06:34:17Z

run buildall

doris-robot · 2025-12-09T08:20:47Z

TPC-DS: Total hot run time: 182820 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit d58dc0f6dd1fac5f4fa6ac60bfd63eb53d3809d8, data reload: false
query5	4556	620	484	484
query6	360	226	216	216
query7	4646	465	281	281
query8	318	255	264	255
query9	8769	2630	2638	2630
query10	570	384	328	328
query11	15352	14955	14595	14595
query12	182	120	112	112
query13	1671	518	378	378
query14	5913	3278	3037	3037
query14_1	2834	2876	2972	2876
query15	202	197	189	189
query16	7269	482	442	442
query17	1239	751	560	560
query18	1969	421	339	339
query19	236	220	197	197
query20	121	112	114	112
query21	221	134	115	115
query22	3866	4041	4166	4041
query23	17129	16704	16463	16463
query23_1	16474	16422	16630	16422
query24	7693	1677	1292	1292
query24_1	1296	1282	1273	1273
query25	611	594	433	433
query26	1403	283	159	159
query27	3009	501	329	329
query28	5027	2223	2269	2223
query29	949	653	506	506
query30	329	263	222	222
query31	877	748	683	683
query32	86	74	75	74
query33	706	363	342	342
query34	1225	942	570	570
query35	865	862	772	772
query36	936	924	854	854
query37	136	97	79	79
query38	4184	4283	3894	3894
query39	758	751	729	729
query39_1	697	707	729	707
query40	239	137	124	124
query41	74	65	64	64
query42	128	109	104	104
query43	450	419	393	393
query44	1303	781	763	763
query45	199	190	196	190
query46	969	976	627	627
query47	1690	1724	1632	1632
query48	423	334	249	249
query49	765	439	367	367
query50	661	308	230	230
query51	3963	3913	3866	3866
query52	129	111	101	101
query53	331	352	300	300
query54	363	270	258	258
query55	98	90	74	74
query56	333	317	315	315
query57	1148	1158	1085	1085
query58	293	272	253	253
query59	2374	2429	2321	2321
query60	368	323	316	316
query61	195	185	186	185
query62	815	677	643	643
query63	346	299	296	296
query64	4526	1288	1011	1011
query65	4045	3940	3950	3940
query66	1236	429	317	317
query67	15535	15125	14735	14735
query68	8262	1018	734	734
query69	576	341	312	312
query70	1123	973	1014	973
query71	505	317	286	286
query72	6009	4923	4892	4892
query73	698	557	305	305
query74	8825	8768	8745	8745
query75	3685	3522	3165	3165
query76	3632	1136	760	760
query77	808	403	300	300
query78	9485	9577	8816	8816
query79	1663	911	629	629
query80	730	649	560	560
query81	493	273	240	240
query82	445	132	106	106
query83	256	256	238	238
query84	251	127	95	95
query85	950	500	467	467
query86	338	302	270	270
query87	4015	4085	3978	3978
query88	3971	2303	2257	2257
query89	492	421	388	388
query90	2056	153	156	153
query91	183	166	146	146
query92	82	68	64	64
query93	1176	897	572	572
query94	761	291	282	282
query95	569	375	325	325
query96	585	484	210	210
query97	2631	2656	2545	2545
query98	242	198	189	189
query99	1351	1288	1230	1230
Total cold run time: 274529 ms
Total hot run time: 182820 ms

doris-robot · 2025-12-09T08:25:49Z

ClickBench: Total hot run time: 27.33 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit d58dc0f6dd1fac5f4fa6ac60bfd63eb53d3809d8, data reload: false
query1	0.05	0.04	0.04
query2	0.10	0.05	0.05
query3	0.25	0.09	0.09
query4	1.61	0.11	0.11
query5	0.28	0.25	0.26
query6	1.20	0.64	0.63
query7	0.03	0.03	0.03
query8	0.06	0.04	0.05
query9	0.57	0.51	0.50
query10	0.56	0.56	0.54
query11	0.15	0.11	0.12
query12	0.15	0.11	0.12
query13	0.63	0.61	0.60
query14	1.00	1.00	0.99
query15	0.82	0.80	0.81
query16	0.43	0.44	0.41
query17	1.01	1.02	1.01
query18	0.23	0.23	0.22
query19	1.85	1.80	1.89
query20	0.02	0.01	0.01
query21	15.44	0.30	0.14
query22	4.94	0.06	0.04
query23	16.07	0.28	0.11
query24	0.97	0.24	0.57
query25	0.09	0.05	0.06
query26	0.14	0.14	0.14
query27	0.05	0.05	0.05
query28	3.80	1.23	1.02
query29	12.59	3.97	3.17
query30	0.27	0.14	0.11
query31	2.82	0.63	0.40
query32	3.23	0.55	0.45
query33	3.02	3.12	3.06
query34	16.86	5.23	4.58
query35	4.56	4.56	4.56
query36	0.64	0.49	0.50
query37	0.11	0.07	0.07
query38	0.08	0.04	0.04
query39	0.05	0.03	0.03
query40	0.17	0.15	0.15
query41	0.09	0.03	0.03
query42	0.04	0.03	0.03
query43	0.04	0.03	0.04
Total cold run time: 97.07 s
Total hot run time: 27.33 s

doris-robot · 2025-12-09T15:21:06Z

BE UT Coverage Report

Increment line coverage 72.41% (63/87) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	53.39% (18726/35073)
Line Coverage	39.13% (173354/443034)
Region Coverage	33.78% (134348/397760)
Branch Coverage	34.68% (57720/166438)

hello-stephen · 2025-12-09T15:27:03Z

BE Regression && UT Coverage Report

Increment line coverage 79.31% (69/87) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	72.20% (24820/34376)
Line Coverage	58.97% (260935/442490)
Region Coverage	53.90% (216934/402467)
Branch Coverage	55.45% (92775/167301)

github-actions · 2025-12-10T12:40:51Z

PR approved by anyone and no changes requested.

github-actions · 2025-12-12T15:08:47Z

PR approved by at least one committer and no changes requested.

…files (apache#58839) Related PR: apache#57770 Problem Summary: The main change is to offload the entire delete bitmap calculation process—including file closing, rowset building, segment loading, and bitmap calculation—to a background thread pool. This prevents blocking the memtable flush thread, enhancing performance and concurrency.

[opt](cloud) optimize load performance for mow table when pack small …
993ad29
…files

liaoxin01 requested a review from gavinchou as a code owner December 9, 2025 03:05

CopilotAI review requested due to automatic review settings December 9, 2025 03:05

liaoxin01 requested a review from dataroaring as a code owner December 9, 2025 03:05

Copilotstarted reviewing on behalf of liaoxin01 December 9, 2025 03:17 View session

CopilotAI reviewed Dec 9, 2025

dataroaring requested a review from mymeiyi December 9, 2025 05:58

bobhan1 reviewed Dec 9, 2025
View reviewed changes

Comment threadbe/src/olap/rowset/beta_rowset_writer.cpp

fix comment
d58dc0f

bobhan1 approved these changes Dec 10, 2025
View reviewed changes

github-actionsBot added the reviewed label Dec 10, 2025

luwei16 approved these changes Dec 12, 2025
View reviewed changes

github-actionsBot added the approved Indicates a PR has been approved by one committer. label Dec 12, 2025

gavinchou approved these changes Dec 15, 2025
View reviewed changes

gavinchou merged commit 678d78f into apache:masterDec 15, 2025
31 of 35 checks passed

liaoxin01 added the packed-file label Dec 15, 2025

yiguolei added the dev/4.0.3-merged label Jan 12, 2026

Uh oh!

Conversation

liaoxin01 commented Dec 9, 2025• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What problem does this PR solve?

Release note

Check List (For Author)

Check List (For Reviewer who merge this PR)

Uh oh!

hello-stephen commented Dec 9, 2025

Uh oh!

liaoxin01 commented Dec 9, 2025

Uh oh!

CopilotAI left a comment

Choose a reason for hiding this comment

Uh oh!

doris-robot commented Dec 9, 2025

Uh oh!

doris-robot commented Dec 9, 2025

Uh oh!

doris-robot commented Dec 9, 2025

Uh oh!

hello-stephen commented Dec 9, 2025

BE UT Coverage Report

Uh oh!

Uh oh!

liaoxin01 commented Dec 9, 2025

Uh oh!

doris-robot commented Dec 9, 2025

Uh oh!

doris-robot commented Dec 9, 2025

Uh oh!

doris-robot commented Dec 9, 2025

BE UT Coverage Report

Uh oh!

hello-stephen commented Dec 9, 2025

BE Regression && UT Coverage Report

Uh oh!

github-actionsBot commented Dec 10, 2025

Uh oh!

github-actionsBot commented Dec 12, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

8 participants

liaoxin01 commented Dec 9, 2025•
edited
Loading