[enhance](memtable) support adaptive memtable write buffer size by sollhui · Pull Request #56948 · apache/doris

sollhui · 2025-10-14T07:44:09Z

What problem does this PR solve?

In some cases, the size of segments may be relatively small, which not only affects read performance but may also cause transaction commit failures in cloud mode for fdb value limit.

We introduce the adaptive memtable write buffer size scheme, which adaptively increases the write buffer size when there is enough memory to avoid the generation of small segments as much as possible.

Release note

None

Check List (For Author)

Test
- Regression test
- Unit Test
- Manual test (add detailed scripts or steps below)
- No need to test or manual test. Explain why:
  - This is a refactor/code format and no logic has been changed.
  - Previous test can cover this change.
  - No code files have been changed.
  - Other reason
Behavior changed:
- No.
- Yes.
Does this need documentation?
- No.
- Yes.

Check List (For Reviewer who merge this PR)

Confirm the release note
Confirm test cases
Confirm document
Add branch pick label

sollhui · 2025-10-14T07:44:33Z

run buildall

sollhui · 2025-10-14T07:51:01Z

run buildall

doris-robot · 2025-10-14T08:49:48Z

ClickBench: Total hot run time: 30.06 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 18b7b6acd9549fdf87e638ad2d7f119c2245527b, data reload: false
query1	0.05	0.05	0.05
query2	0.10	0.05	0.06
query3	0.25	0.09	0.09
query4	1.63	0.13	0.13
query5	0.29	0.28	0.26
query6	1.27	0.68	0.64
query7	0.03	0.03	0.03
query8	0.06	0.05	0.05
query9	0.64	0.54	0.52
query10	0.60	0.58	0.58
query11	0.19	0.11	0.12
query12	0.15	0.12	0.12
query13	0.63	0.62	0.62
query14	1.01	1.01	1.04
query15	0.87	0.89	0.87
query16	0.40	0.40	0.39
query17	1.03	1.09	1.04
query18	0.21	0.20	0.20
query19	1.91	1.89	1.87
query20	0.02	0.01	0.01
query21	15.42	0.89	0.57
query22	0.77	1.20	0.61
query23	15.01	1.43	0.68
query24	6.85	1.39	0.38
query25	0.53	0.26	0.12
query26	0.62	0.16	0.14
query27	0.07	0.05	0.06
query28	8.87	1.39	0.92
query29	12.60	3.89	3.22
query30	0.28	0.14	0.11
query31	2.82	0.63	0.38
query32	3.24	0.55	0.48
query33	3.14	3.08	3.13
query34	16.03	5.42	4.84
query35	4.92	4.95	4.93
query36	0.68	0.52	0.51
query37	0.11	0.07	0.08
query38	0.07	0.05	0.05
query39	0.04	0.03	0.03
query40	0.18	0.16	0.15
query41	0.09	0.04	0.03
query42	0.04	0.03	0.03
query43	0.04	0.04	0.04
Total cold run time: 103.76 s
Total hot run time: 30.06 s

sollhui · 2025-10-14T09:59:45Z

run buildall

sollhui · 2025-10-14T10:00:09Z

run buildall

doris-robot · 2025-10-14T11:40:53Z

TPC-DS: Total hot run time: 190045 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 41affff5c078c617e19c81ad1b586e085ba12fd6, data reload: false
query1	1095	436	416	416
query2	6571	1704	1691	1691
query3	6761	221	218	218
query4	26886	23745	23090	23090
query5	5828	644	498	498
query6	342	255	221	221
query7	4652	510	296	296
query8	323	284	274	274
query9	8708	2560	2547	2547
query10	583	326	297	297
query11	15342	15172	14973	14973
query12	184	128	118	118
query13	1718	560	458	458
query14	12635	9195	9201	9195
query15	254	192	175	175
query16	7817	677	500	500
query17	1571	788	705	705
query18	2125	481	366	366
query19	302	226	176	176
query20	145	136	138	136
query21	237	148	118	118
query22	4804	4919	4576	4576
query23	34661	33825	34070	33825
query24	8359	2466	2527	2466
query25	620	568	511	511
query26	1734	285	163	163
query27	2829	527	373	373
query28	4392	2245	2216	2216
query29	800	653	549	549
query30	356	237	210	210
query31	897	922	774	774
query32	89	74	68	68
query33	592	391	347	347
query34	834	872	524	524
query35	832	858	795	795
query36	1013	1035	935	935
query37	121	110	95	95
query38	3729	3776	3718	3718
query39	1565	1467	1496	1467
query40	226	135	139	135
query41	73	61	61	61
query42	119	108	111	108
query43	487	496	472	472
query44	1314	830	829	829
query45	193	187	176	176
query46	844	1002	645	645
query47	1812	1890	1731	1731
query48	379	410	310	310
query49	743	493	407	407
query50	638	683	407	407
query51	3915	3955	3869	3869
query52	113	105	105	105
query53	226	260	202	202
query54	600	579	512	512
query55	91	83	85	83
query56	310	331	320	320
query57	1176	1201	1142	1142
query58	286	276	272	272
query59	2511	2591	2515	2515
query60	342	337	332	332
query61	152	149	200	149
query62	812	712	656	656
query63	229	195	185	185
query64	4409	1169	847	847
query65	4036	3945	3951	3945
query66	1037	415	329	329
query67	15546	15509	15176	15176
query68	6391	948	598	598
query69	506	323	285	285
query70	1371	1291	1311	1291
query71	497	347	305	305
query72	5850	4929	4869	4869
query73	526	591	354	354
query74	9425	8937	8498	8498
query75	3757	3414	2813	2813
query76	3209	1158	735	735
query77	796	398	326	326
query78	9518	9792	8942	8942
query79	2747	799	606	606
query80	668	571	469	469
query81	523	271	229	229
query82	224	161	139	139
query83	280	260	242	242
query84	256	109	102	102
query85	875	481	405	405
query86	341	305	284	284
query87	3780	3763	3653	3653
query88	4104	2209	2206	2206
query89	377	324	293	293
query90	2038	211	215	211
query91	158	169	133	133
query92	88	67	65	65
query93	2323	971	643	643
query94	639	438	349	349
query95	396	321	309	309
query96	489	573	279	279
query97	2924	2984	2867	2867
query98	251	214	214	214
query99	1351	1401	1331	1331
Total cold run time: 281603 ms
Total hot run time: 190045 ms

doris-robot · 2025-10-14T11:46:23Z

ClickBench: Total hot run time: 30.63 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 41affff5c078c617e19c81ad1b586e085ba12fd6, data reload: false
query1	0.05	0.05	0.04
query2	0.10	0.06	0.05
query3	0.25	0.09	0.08
query4	1.60	0.12	0.11
query5	0.27	0.26	0.25
query6	1.18	0.66	0.63
query7	0.03	0.03	0.02
query8	0.06	0.04	0.05
query9	0.62	0.52	0.52
query10	0.57	0.61	0.57
query11	0.16	0.11	0.11
query12	0.15	0.12	0.12
query13	0.64	0.61	0.61
query14	1.00	1.02	1.02
query15	0.86	0.85	0.86
query16	0.39	0.40	0.39
query17	1.04	1.05	1.06
query18	0.22	0.20	0.20
query19	1.98	1.90	1.82
query20	0.01	0.01	0.01
query21	15.46	0.93	0.58
query22	0.76	1.07	0.63
query23	15.08	1.38	0.69
query24	6.74	1.52	1.08
query25	0.48	0.11	0.15
query26	0.63	0.17	0.13
query27	0.06	0.06	0.05
query28	9.90	1.34	0.95
query29	12.60	4.01	3.30
query30	0.28	0.15	0.11
query31	2.83	0.60	0.38
query32	3.25	0.57	0.47
query33	3.10	3.07	3.14
query34	16.15	5.47	4.83
query35	4.88	4.93	4.88
query36	0.69	0.52	0.50
query37	0.10	0.08	0.07
query38	0.08	0.05	0.04
query39	0.03	0.03	0.03
query40	0.18	0.16	0.15
query41	0.08	0.04	0.03
query42	0.04	0.03	0.03
query43	0.05	0.03	0.05
Total cold run time: 104.63 s
Total hot run time: 30.63 s

dataroaring

LGTM

github-actions · 2025-10-14T12:51:06Z

PR approved by at least one committer and no changes requested.

github-actions · 2025-10-14T12:51:10Z

PR approved by anyone and no changes requested.

hello-stephen · 2025-10-14T13:18:20Z

BE UT Coverage Report

Increment line coverage 66.67% (10/15) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	52.55% (17755/33790)
Line Coverage	37.75% (161468/427761)
Region Coverage	32.20% (123179/382593)
Branch Coverage	33.63% (54121/160935)

hello-stephen · 2025-10-14T14:23:24Z

BE Regression && UT Coverage Report

Increment line coverage 66.67% (10/15) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	71.14% (23565/33125)
Line Coverage	57.56% (246066/427507)
Region Coverage	52.52% (203549/387543)
Branch Coverage	54.46% (88135/161824)

sollhui · 2025-10-15T08:09:39Z

run buildall

doris-robot · 2025-10-15T09:47:05Z

TPC-DS: Total hot run time: 188340 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 697085113e3c0fa673b1f65c0c05c126551c7f66, data reload: false
query1	1075	422	411	411
query2	6135	1724	1739	1724
query3	4262	222	226	222
query4	25767	23463	23029	23029
query5	4546	640	486	486
query6	314	237	228	228
query7	4326	502	306	306
query8	307	259	273	259
query9	6778	2595	2561	2561
query10	508	334	302	302
query11	15537	15282	14762	14762
query12	195	115	112	112
query13	1649	549	428	428
query14	11233	9318	9333	9318
query15	216	204	201	201
query16	7692	701	531	531
query17	1606	798	676	676
query18	2022	426	319	319
query19	215	205	168	168
query20	138	126	119	119
query21	217	145	125	125
query22	4113	4134	4127	4127
query23	34047	33190	32989	32989
query24	7988	2391	2385	2385
query25	582	499	444	444
query26	1239	271	160	160
query27	2729	499	355	355
query28	4416	2163	2179	2163
query29	809	613	478	478
query30	309	232	214	214
query31	911	822	756	756
query32	86	88	75	75
query33	589	378	326	326
query34	855	859	531	531
query35	803	891	765	765
query36	983	999	929	929
query37	123	108	95	95
query38	3566	3502	3454	3454
query39	1478	1461	1482	1461
query40	221	129	123	123
query41	65	62	63	62
query42	128	116	110	110
query43	500	509	483	483
query44	1386	850	814	814
query45	195	185	179	179
query46	883	1004	637	637
query47	1786	1791	1709	1709
query48	401	420	326	326
query49	774	532	417	417
query50	668	696	414	414
query51	3918	3857	3860	3857
query52	120	111	99	99
query53	251	283	207	207
query54	599	597	548	548
query55	94	90	88	88
query56	345	340	324	324
query57	1182	1230	1110	1110
query58	289	303	294	294
query59	2646	2684	2519	2519
query60	356	347	330	330
query61	160	150	165	150
query62	789	733	677	677
query63	239	199	205	199
query64	4519	1157	828	828
query65	4054	3958	3944	3944
query66	1081	517	357	357
query67	15407	15189	14994	14994
query68	8485	949	595	595
query69	533	330	304	304
query70	1412	1292	1282	1282
query71	460	346	322	322
query72	5755	4957	4884	4884
query73	600	568	362	362
query74	8862	8823	8880	8823
query75	3355	3334	2812	2812
query76	3571	1168	765	765
query77	483	408	344	344
query78	9743	9626	8835	8835
query79	2608	838	595	595
query80	631	565	500	500
query81	502	279	256	256
query82	254	236	149	149
query83	265	269	244	244
query84	257	113	96	96
query85	885	469	429	429
query86	379	304	323	304
query87	3704	3756	3641	3641
query88	3488	2296	2292	2292
query89	399	332	308	308
query90	1990	230	227	227
query91	244	167	140	140
query92	88	76	69	69
query93	1970	1006	655	655
query94	765	445	339	339
query95	404	328	312	312
query96	499	588	289	289
query97	2960	2991	2869	2869
query98	246	207	213	207
query99	1339	1422	1298	1298
Total cold run time: 270556 ms
Total hot run time: 188340 ms

doris-robot · 2025-10-15T09:52:33Z

ClickBench: Total hot run time: 29.9 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 697085113e3c0fa673b1f65c0c05c126551c7f66, data reload: false
query1	0.05	0.05	0.05
query2	0.10	0.05	0.06
query3	0.25	0.08	0.08
query4	1.61	0.12	0.12
query5	0.27	0.28	0.24
query6	1.19	0.64	0.64
query7	0.02	0.03	0.03
query8	0.05	0.04	0.04
query9	0.62	0.53	0.53
query10	0.58	0.60	0.58
query11	0.17	0.11	0.11
query12	0.16	0.12	0.12
query13	0.64	0.63	0.62
query14	1.02	1.04	1.04
query15	0.90	0.86	0.84
query16	0.40	0.40	0.43
query17	1.05	1.08	1.05
query18	0.22	0.20	0.20
query19	1.93	1.80	1.85
query20	0.02	0.01	0.02
query21	15.47	0.92	0.59
query22	0.76	1.22	0.62
query23	14.92	1.39	0.62
query24	7.00	1.20	0.36
query25	0.32	0.19	0.11
query26	0.58	0.16	0.13
query27	0.08	0.06	0.06
query28	9.32	1.37	0.92
query29	12.58	3.89	3.24
query30	0.29	0.15	0.11
query31	2.84	0.59	0.39
query32	3.26	0.58	0.48
query33	3.12	3.10	3.09
query34	16.26	5.50	4.86
query35	5.00	4.87	4.93
query36	0.71	0.52	0.52
query37	0.10	0.08	0.07
query38	0.06	0.05	0.04
query39	0.03	0.03	0.03
query40	0.19	0.16	0.14
query41	0.09	0.04	0.03
query42	0.04	0.03	0.03
query43	0.04	0.04	0.04
Total cold run time: 104.31 s
Total hot run time: 29.9 s

hello-stephen · 2025-10-15T11:37:33Z

BE UT Coverage Report

Increment line coverage 48.39% (15/31) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	52.57% (17789/33837)
Line Coverage	37.77% (161665/428049)
Region Coverage	32.20% (123249/382719)
Branch Coverage	33.63% (54149/161015)

hello-stephen · 2025-10-15T12:40:28Z

BE Regression && UT Coverage Report

Increment line coverage 70.97% (22/31) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	71.20% (23615/33169)
Line Coverage	57.62% (246477/427792)
Region Coverage	52.69% (204265/387666)
Branch Coverage	54.53% (88281/161904)

liaoxin01

LGTM

github-actions · 2025-10-16T01:27:00Z

PR approved by at least one committer and no changes requested.

### What problem does this PR solve? In some cases, the size of `segments` may be relatively small, which not only affects read performance but may also cause transaction commit failures in cloud mode for fdb value limit. We introduce the adaptive memtable write buffer size scheme, which adaptively increases the write buffer size when there is enough memory to avoid the generation of small segments as much as possible. ### Release note None ### Check List (For Author) - Test  - [ ] Regression test - [ ] Unit Test - [ ] Manual test (add detailed scripts or steps below) - [ ] No need to test or manual test. Explain why: - [ ] This is a refactor/code format and no logic has been changed. - [ ] Previous test can cover this change. - [ ] No code files have been changed. - [ ] Other reason  - Behavior changed: - [ ] No. - [ ] Yes.  - Does this need documentation? - [ ] No. - [ ] Yes.  ### Check List (For Reviewer who merge this PR) - [ ] Confirm the release note - [ ] Confirm test cases - [ ] Confirm document - [ ] Add branch pick label

…r size #56948 (#57023) Cherry-picked from #56948 Co-authored-by: hui lai <laihui@selectdb.com>

…ze (apache#56948)" This reverts commit 0a2b8d8.

…te buffer size (#56948)" (#59926) This reverts commit 0a2b8d8. ### What problem does this PR solve? Issue Number: close #xxx Related PR: #xxx Problem Summary: ### Release note None ### Check List (For Author) - Test  - [ ] Regression test - [ ] Unit Test - [ ] Manual test (add detailed scripts or steps below) - [ ] No need to test or manual test. Explain why: - [ ] This is a refactor/code format and no logic has been changed. - [ ] Previous test can cover this change. - [ ] No code files have been changed. - [ ] Other reason  - Behavior changed: - [ ] No. - [ ] Yes.  - Does this need documentation? - [ ] No. - [ ] Yes.  ### Check List (For Reviewer who merge this PR) - [ ] Confirm the release note - [ ] Confirm test cases - [ ] Confirm document - [ ] Add branch pick label

…te buffer size (apache#56948)" (apache#59926) This reverts commit 0a2b8d8. ### What problem does this PR solve? Issue Number: close #xxx Related PR: #xxx Problem Summary: ### Release note None ### Check List (For Author) - Test  - [ ] Regression test - [ ] Unit Test - [ ] Manual test (add detailed scripts or steps below) - [ ] No need to test or manual test. Explain why: - [ ] This is a refactor/code format and no logic has been changed. - [ ] Previous test can cover this change. - [ ] No code files have been changed. - [ ] Other reason  - Behavior changed: - [ ] No. - [ ] Yes.  - Does this need documentation? - [ ] No. - [ ] Yes.  ### Check List (For Reviewer who merge this PR) - [ ] Confirm the release note - [ ] Confirm test cases - [ ] Confirm document - [ ] Add branch pick label

…61810) pick #56948 #62104

…62308) pick #56948 #62104

sollhui force-pushed the adaptive_write_buffer_size branch from c1414e9 to 18b7b6aCompare October 14, 2025 07:50

dataroaring reviewed Oct 14, 2025
View reviewed changes

Comment threadbe/src/olap/memtable.h Outdated

dataroaring added dev/4.0.x dev/3.1.x labels Oct 14, 2025

sollhui force-pushed the adaptive_write_buffer_size branch from 18b7b6a to 73a5aa0Compare October 14, 2025 09:33

sollhui force-pushed the adaptive_write_buffer_size branch from 73a5aa0 to 41affffCompare October 14, 2025 10:00

dataroaring previously approved these changes Oct 14, 2025
View reviewed changes

0AyanamiRei approved these changes Oct 14, 2025
View reviewed changes

github-actionsBot added approved Indicates a PR has been approved by one committer. reviewed labels Oct 14, 2025

dataroaring removed the dev/3.1.x label Oct 15, 2025

support adaptive write buffer size
6970851

sollhui dismissed dataroaring’s stale review via 6970851October 15, 2025 08:09

sollhui force-pushed the adaptive_write_buffer_size branch from 41affff to 6970851Compare October 15, 2025 08:09

github-actionsBot removed the approved Indicates a PR has been approved by one committer. label Oct 15, 2025

liaoxin01 approved these changes Oct 16, 2025
View reviewed changes

github-actionsBot added the approved Indicates a PR has been approved by one committer. label Oct 16, 2025

dataroaring merged commit 0a2b8d8 into apache:masterOct 16, 2025
26 of 27 checks passed

github-actionsBot mentioned this pull request Oct 16, 2025
branch-4.0: [enhance](memtable) support adaptive memtable write buffer size #56948 #57023
Merged

yiguolei pushed a commit that referenced this pull request Oct 16, 2025
branch-4.0: [enhance](memtable) support adaptive memtable write buffe…
b153d7e
…r size #56948 (#57023) Cherry-picked from #56948 Co-authored-by: hui lai <laihui@selectdb.com>

yiguolei added dev/4.0.1-merged and removed dev/4.0.x labels Oct 16, 2025

sollhui added a commit to sollhui/doris that referenced this pull request Jan 15, 2026
Revert "[enhance](memtable) support adaptive memtable write buffer si…
2ff88bd
…ze (apache#56948)" This reverts commit 0a2b8d8.

sollhui mentioned this pull request Mar 27, 2026
branch-4.0: [enhance](memtable) support adaptive write buffer size #61810
Merged

liaoxin01 added the dev/4.1.x label Mar 30, 2026

github-actionsBot added the dev/4.1.x-conflict label Mar 30, 2026

sollhui mentioned this pull request Apr 10, 2026
branch-4.1: [enhance](memtable) support adaptive write buffer size #62308
Merged

yiguolei pushed a commit that referenced this pull request Apr 27, 2026
branch-4.0: [enhance](memtable) support adaptive write buffer size (#…
07e71c6
…61810) pick #56948 #62104

yiguolei pushed a commit that referenced this pull request Apr 27, 2026
branch-4.1: [enhance](memtable) support adaptive write buffer size (#…
3eb35ed
…62308) pick #56948 #62104

yiguolei added dev/4.1.1-merged and removed dev/4.1.x dev/4.1.x-conflict labels Apr 27, 2026

yiguolei mentioned this pull request May 20, 2026
4.1.1 Release Notes #63426
Closed

Uh oh!

Conversation

sollhui commented Oct 14, 2025

What problem does this PR solve?

Release note

Check List (For Author)

Check List (For Reviewer who merge this PR)

Uh oh!

sollhui commented Oct 14, 2025

Uh oh!

sollhui commented Oct 14, 2025

Uh oh!

Uh oh!

doris-robot commented Oct 14, 2025

Uh oh!

sollhui commented Oct 14, 2025

Uh oh!

sollhui commented Oct 14, 2025

Uh oh!

doris-robot commented Oct 14, 2025

Uh oh!

doris-robot commented Oct 14, 2025

Uh oh!

dataroaring left a comment

Choose a reason for hiding this comment

Uh oh!

github-actionsBot commented Oct 14, 2025

Uh oh!

github-actionsBot commented Oct 14, 2025

Uh oh!

hello-stephen commented Oct 14, 2025

BE UT Coverage Report

Uh oh!

hello-stephen commented Oct 14, 2025

BE Regression && UT Coverage Report

Uh oh!

sollhui commented Oct 15, 2025

Uh oh!

doris-robot commented Oct 15, 2025

Uh oh!

doris-robot commented Oct 15, 2025

Uh oh!

hello-stephen commented Oct 15, 2025

BE UT Coverage Report

Uh oh!

hello-stephen commented Oct 15, 2025

BE Regression && UT Coverage Report

Uh oh!

liaoxin01 left a comment

Choose a reason for hiding this comment

Uh oh!

github-actionsBot commented Oct 16, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants