[feature](cloud) Support compaction read-write separation for cloud mode by Yukang-Lian · Pull Request #60310 · apache/doris

Yukang-Lian · 2026-01-28T09:46:04Z

What problem does this PR solve?

Issue Number: close #xxx

Related PR: #xxx

Problem Summary:

In cloud mode with multiple compute groups sharing the same tablets, compaction executed by read-only clusters invalidates file cache on the write cluster and wastes resources. This PR implements compaction read-write separation so that only the cluster that last performed data load executes compaction.

Release note

None

Check List (For Author)

Test
- Regression test
- Unit Test
- Manual test (add detailed scripts or steps below)
- No need to test or manual test. Explain why:
  - This is a refactor/code format and no logic has been changed.
  - Previous test can cover this change.
  - No code files have been changed.
  - Other reason
Behavior changed:
- No.
- Yes.
Does this need documentation?
- No.
- Yes.

Check List (For Reviewer who merge this PR)

Confirm the release note
Confirm test cases
Confirm document
Add branch pick label

Thearas · 2026-01-28T09:46:10Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

What problem was fixed (it's best to include specific error reporting information). How it was fixed.
Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
What features were added. Why was this function added?
Which code was refactored and why was this part of the code refactored?
Which functions were optimized and what is the difference before and after the optimization?

Yukang-Lian · 2026-01-29T14:41:16Z

run buildall

hello-stephen · 2026-01-29T16:18:30Z

FE UT Coverage Report

Increment line coverage `` 🎉
Increment coverage report
Complete coverage report

Yukang-Lian · 2026-01-29T17:15:19Z

run buildall

doris-robot · 2026-01-29T19:05:56Z

TPC-H: Total hot run time: 32338 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 45e7d1fbb71cfe904b139878d2e4f80352f77f1a, data reload: false
------ Round 1 ----------------------------------
q1	17652	5282	5080	5080
q2	1997	321	211	211
q3	10202	1310	745	745
q4	10205	840	311	311
q5	7487	2148	1946	1946
q6	204	182	147	147
q7	882	755	607	607
q8	9282	1395	1111	1111
q9	5560	4853	4793	4793
q10	6904	1965	1562	1562
q11	550	290	286	286
q12	398	377	227	227
q13	17802	4059	3230	3230
q14	240	235	209	209
q15	916	833	823	823
q16	661	685	621	621
q17	834	842	424	424
q18	7501	6722	7298	6722
q19	1607	1041	635	635
q20	422	370	244	244
q21	3022	2219	2087	2087
q22	359	384	317	317
Total cold run time: 104687 ms
Total hot run time: 32338 ms
----- Round 2, with runtime_filter_mode=off -----
q1	5723	5614	5616	5614
q2	274	332	244	244
q3	2371	2894	2473	2473
q4	1466	1889	1386	1386
q5	4744	4629	4564	4564
q6	221	187	140	140
q7	2067	1980	1790	1790
q8	2571	2348	2454	2348
q9	7540	7470	7375	7375
q10	2799	3050	2681	2681
q11	551	473	448	448
q12	638	756	629	629
q13	3924	4072	3212	3212
q14	282	282	269	269
q15	846	814	802	802
q16	639	686	645	645
q17	1084	1244	1267	1244
q18	7573	7204	7314	7204
q19	831	795	832	795
q20	1981	2033	1890	1890
q21	4535	4100	4112	4100
q22	600	560	495	495
Total cold run time: 53260 ms
Total hot run time: 50348 ms

doris-robot · 2026-01-29T19:22:45Z

ClickBench: Total hot run time: 28.55 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 45e7d1fbb71cfe904b139878d2e4f80352f77f1a, data reload: false
query1	0.06	0.05	0.05
query2	0.09	0.04	0.05
query3	0.25	0.08	0.08
query4	1.60	0.11	0.11
query5	0.27	0.25	0.24
query6	1.16	0.67	0.66
query7	0.04	0.02	0.02
query8	0.04	0.04	0.04
query9	0.57	0.50	0.50
query10	0.57	0.54	0.55
query11	0.14	0.09	0.09
query12	0.13	0.10	0.11
query13	0.62	0.62	0.61
query14	1.06	1.07	1.05
query15	0.88	0.85	0.86
query16	0.38	0.39	0.38
query17	1.17	1.10	1.07
query18	0.22	0.20	0.21
query19	2.06	2.00	1.98
query20	0.02	0.02	0.02
query21	15.41	0.28	0.15
query22	4.91	0.06	0.05
query23	15.91	0.30	0.10
query24	0.90	0.76	0.31
query25	0.09	0.06	0.07
query26	0.14	0.13	0.14
query27	0.08	0.05	0.06
query28	3.56	1.18	0.96
query29	12.55	3.88	3.13
query30	0.28	0.14	0.11
query31	2.81	0.64	0.39
query32	3.27	0.60	0.50
query33	3.18	3.24	3.27
query34	16.64	5.38	4.70
query35	4.76	5.14	5.41
query36	0.68	0.56	0.55
query37	0.11	0.08	0.06
query38	0.07	0.04	0.04
query39	0.04	0.03	0.03
query40	0.22	0.17	0.15
query41	0.08	0.02	0.02
query42	0.04	0.02	0.02
query43	0.05	0.04	0.03
Total cold run time: 97.11 s
Total hot run time: 28.55 s

hello-stephen · 2026-01-29T20:41:49Z

BE UT Coverage Report

Increment line coverage 7.98% (15/188) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	52.74% (19279/36558)
Line Coverage	36.12% (179160/495960)
Region Coverage	32.55% (138956/426866)
Branch Coverage	33.52% (60133/179421)

hello-stephen · 2026-01-29T22:19:57Z

BE Regression && UT Coverage Report

Increment line coverage 43.62% (82/188) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	71.49% (25620/35836)
Line Coverage	54.12% (267758/494727)
Region Coverage	51.71% (222948/431184)
Branch Coverage	53.15% (95741/180137)

Yukang-Lian · 2026-01-30T04:09:10Z

run buildall

hello-stephen · 2026-01-30T05:20:37Z

FE UT Coverage Report

Increment line coverage `` 🎉
Increment coverage report
Complete coverage report

Yukang-Lian · 2026-01-30T07:50:34Z

run buildall

hello-stephen · 2026-01-30T09:05:05Z

FE UT Coverage Report

Increment line coverage `` 🎉
Increment coverage report
Complete coverage report

doris-robot · 2026-01-30T09:42:41Z

TPC-H: Total hot run time: 31948 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 006d993f151aa35a8607afc0d9941d132aa8033e, data reload: false
------ Round 1 ----------------------------------
q1	17643	5268	5067	5067
q2	2043	316	200	200
q3	10193	1313	767	767
q4	10211	889	327	327
q5	7530	2137	1947	1947
q6	196	179	148	148
q7	894	740	608	608
q8	9254	1421	1145	1145
q9	5419	4884	4904	4884
q10	6912	1974	1568	1568
q11	536	295	279	279
q12	378	387	236	236
q13	17770	4080	3218	3218
q14	249	253	227	227
q15	920	850	807	807
q16	695	677	641	641
q17	635	797	505	505
q18	7090	6506	6371	6371
q19	1442	995	614	614
q20	390	356	237	237
q21	2720	2086	1875	1875
q22	374	324	277	277
Total cold run time: 103494 ms
Total hot run time: 31948 ms
----- Round 2, with runtime_filter_mode=off -----
q1	5409	5295	5316	5295
q2	260	342	256	256
q3	2187	2668	2245	2245
q4	1356	1724	1308	1308
q5	4258	4194	4304	4194
q6	220	188	145	145
q7	2419	2043	1845	1845
q8	2582	2441	2429	2429
q9	7589	7357	7624	7357
q10	2820	2979	2653	2653
q11	614	478	476	476
q12	742	764	644	644
q13	3904	4381	3539	3539
q14	323	338	309	309
q15	863	820	872	820
q16	683	731	677	677
q17	1213	1651	1398	1398
q18	8254	7783	7699	7699
q19	926	889	989	889
q20	2121	2251	2053	2053
q21	4814	4386	4139	4139
q22	586	549	540	540
Total cold run time: 54143 ms
Total hot run time: 50910 ms

doris-robot · 2026-01-30T09:59:36Z

ClickBench: Total hot run time: 28.89 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 006d993f151aa35a8607afc0d9941d132aa8033e, data reload: false
query1	0.05	0.05	0.05
query2	0.11	0.04	0.04
query3	0.26	0.08	0.09
query4	1.61	0.12	0.11
query5	0.26	0.23	0.24
query6	1.18	0.68	0.66
query7	0.03	0.02	0.03
query8	0.04	0.04	0.04
query9	0.57	0.51	0.50
query10	0.53	0.54	0.55
query11	0.15	0.09	0.10
query12	0.14	0.10	0.10
query13	0.62	0.62	0.61
query14	1.06	1.06	1.07
query15	0.87	0.85	0.86
query16	0.42	0.39	0.40
query17	1.08	1.15	1.17
query18	0.23	0.22	0.22
query19	2.09	2.05	2.05
query20	0.02	0.01	0.01
query21	15.60	0.26	0.14
query22	4.73	0.06	0.05
query23	15.78	0.29	0.10
query24	1.77	0.22	0.41
query25	0.10	0.06	0.06
query26	0.16	0.13	0.13
query27	0.07	0.06	0.06
query28	4.37	1.15	0.97
query29	12.59	3.93	3.17
query30	0.27	0.13	0.11
query31	2.81	0.63	0.41
query32	3.24	0.59	0.51
query33	3.19	3.24	3.34
query34	15.98	5.37	4.71
query35	4.80	5.36	5.31
query36	0.70	0.57	0.53
query37	0.14	0.07	0.07
query38	0.09	0.04	0.04
query39	0.05	0.02	0.03
query40	0.20	0.18	0.17
query41	0.08	0.03	0.02
query42	0.05	0.02	0.03
query43	0.04	0.03	0.04
Total cold run time: 98.13 s
Total hot run time: 28.89 s

doris-robot · 2026-01-30T11:05:28Z

BE UT Coverage Report

Increment line coverage 53.72% (101/188) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	52.50% (19284/36732)
Line Coverage	35.98% (179184/497968)
Region Coverage	32.43% (139235/429311)
Branch Coverage	33.34% (60124/180347)

hello-stephen · 2026-01-30T14:13:18Z

BE Regression && UT Coverage Report

Increment line coverage 84.09% (148/176) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	71.54% (25759/36004)
Line Coverage	54.22% (269367/496779)
Region Coverage	51.94% (225267/433718)
Branch Coverage	53.24% (96403/181079)

Yukang-Lian · 2026-02-03T09:30:10Z

run buildall

doris-robot · 2026-02-03T10:06:58Z

Cloud UT Coverage Report

Increment line coverage 68.89% (62/90) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	79.30% (1793/2261)
Line Coverage	64.76% (31896/49251)
Region Coverage	65.41% (15923/24344)
Branch Coverage	55.95% (8467/15134)

doris-robot · 2026-02-03T11:38:25Z

TPC-H: Total hot run time: 32217 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 159ab4d7cbe585e36ff37f1f25f93c83b22ff4b5, data reload: false
------ Round 1 ----------------------------------
q1	17627	5273	5128	5128
q2	2054	309	199	199
q3	10609	1328	747	747
q4	10319	802	321	321
q5	9573	2139	1970	1970
q6	246	183	150	150
q7	936	744	596	596
q8	9268	1392	1076	1076
q9	5334	4853	4872	4853
q10	6856	1970	1571	1571
q11	525	286	275	275
q12	410	373	223	223
q13	17802	4081	3268	3268
q14	230	237	229	229
q15	928	830	808	808
q16	659	682	628	628
q17	659	803	503	503
q18	6935	6578	6542	6542
q19	1147	996	625	625
q20	400	347	232	232
q21	2623	2076	2003	2003
q22	357	323	270	270
Total cold run time: 105497 ms
Total hot run time: 32217 ms
----- Round 2, with runtime_filter_mode=off -----
q1	5397	5327	5337	5327
q2	256	331	240	240
q3	2163	2677	2264	2264
q4	1338	1714	1315	1315
q5	4295	4215	4817	4215
q6	237	200	159	159
q7	2166	1948	1875	1875
q8	2595	2506	2385	2385
q9	7580	7399	7517	7399
q10	2886	3077	2614	2614
q11	562	488	449	449
q12	765	753	616	616
q13	4048	4252	3563	3563
q14	327	331	310	310
q15	997	904	856	856
q16	693	735	678	678
q17	1184	1361	1640	1361
q18	8007	7832	7792	7792
q19	915	874	865	865
q20	2194	2179	1915	1915
q21	4522	4193	4146	4146
q22	582	548	511	511
Total cold run time: 53709 ms
Total hot run time: 50855 ms

doris-robot · 2026-02-03T11:55:05Z

ClickBench: Total hot run time: 27.96 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 159ab4d7cbe585e36ff37f1f25f93c83b22ff4b5, data reload: false
query1	0.06	0.04	0.04
query2	0.13	0.06	0.06
query3	0.30	0.08	0.07
query4	1.61	0.10	0.10
query5	0.26	0.25	0.24
query6	1.14	0.65	0.63
query7	0.03	0.02	0.02
query8	0.07	0.05	0.06
query9	0.59	0.50	0.50
query10	0.54	0.55	0.56
query11	0.25	0.13	0.13
query12	0.26	0.14	0.15
query13	0.62	0.62	0.61
query14	0.98	0.97	0.98
query15	0.90	0.81	0.82
query16	0.40	0.39	0.40
query17	1.05	0.99	1.05
query18	0.24	0.22	0.22
query19	1.95	1.88	1.88
query20	0.02	0.02	0.01
query21	15.40	0.28	0.24
query22	5.04	0.10	0.09
query23	15.39	0.40	0.22
query24	2.35	0.50	0.34
query25	0.10	0.10	0.10
query26	0.18	0.18	0.17
query27	0.09	0.09	0.09
query28	3.71	1.13	0.96
query29	12.51	4.01	3.27
query30	0.33	0.13	0.11
query31	2.80	0.67	0.44
query32	3.24	0.62	0.49
query33	3.02	3.11	3.08
query34	15.92	5.06	4.43
query35	4.49	4.50	4.54
query36	0.62	0.49	0.50
query37	0.25	0.09	0.09
query38	0.22	0.06	0.06
query39	0.07	0.06	0.05
query40	0.19	0.17	0.16
query41	0.14	0.06	0.07
query42	0.08	0.05	0.05
query43	0.08	0.06	0.06
Total cold run time: 97.62 s
Total hot run time: 27.96 s

hello-stephen · 2026-02-11T16:59:56Z

FE Regression Coverage Report

Increment line coverage 55.56% (5/9) 🎉
Increment coverage report
Complete coverage report

github-actions · 2026-02-12T07:39:02Z

PR approved by at least one committer and no changes requested.

Yukang-Lian · 2026-02-12T08:19:41Z

run external

Yukang-Lian · 2026-02-12T09:24:32Z

run external

hello-stephen · 2026-02-12T11:25:48Z

BE Regression && UT Coverage Report

Increment line coverage 93.78% (181/193) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	71.74% (25950/36174)
Line Coverage	54.36% (271540/499499)
Region Coverage	51.62% (225153/436147)
Branch Coverage	53.19% (96847/182068)

hello-stephen · 2026-02-12T11:33:06Z

FE Regression Coverage Report

Increment line coverage 55.56% (5/9) 🎉
Increment coverage report
Complete coverage report

…ode (apache#60310)

## Proposed changes Cherry-pick the following PRs to branch-4.0: - [Enhancement](Recycler) Add bvar metrics for operation logs #60520 - [Feat](Snapshot) require root privilege for cluster snapshot commands #60239 - [Fix](Recycler) Fix recycler OOM by limiting queued delete tasks #59331 - [feature](cloud) Support compaction read-write separation for cloud mode #60310

## Proposed changes Cherry-pick the following PRs to branch-4.1: - [Enhancement](Recycler) Add bvar metrics for operation logs #60520 - [Feat](Snapshot) require root privilege for cluster snapshot commands #60239 - [feature](cloud) Support compaction read-write separation for cloud mode #60310

Yukang-Lian requested review from dataroaring, gavinchou and w41ter as code owners January 28, 2026 09:46

Yukang-Lian force-pushed the feature/compaction-rw-separation branch from aff26fb to b22ecccCompare January 29, 2026 09:37

Yukang-Lian changed the title ~~[DO NOT MERGE]JUST TEST~~[feature](cloud) Support compaction read-write separation for cloud mode Jan 29, 2026

Yukang-Lian force-pushed the feature/compaction-rw-separation branch from 23e3825 to 45e7d1fCompare January 29, 2026 17:15

Yukang-Lian force-pushed the feature/compaction-rw-separation branch from 45e7d1f to 5d2f93cCompare January 30, 2026 04:04

gavinchou reviewed Jan 30, 2026
View reviewed changes

Comment threadcloud/src/meta-store/keys.h

Yukang-Lian force-pushed the feature/compaction-rw-separation branch from 5d2f93c to 2f6a6d9Compare January 30, 2026 07:48

Yukang-Lian force-pushed the feature/compaction-rw-separation branch from 006d993 to 159ab4dCompare February 3, 2026 09:02

gavinchou approved these changes Feb 12, 2026
View reviewed changes

luwei16 approved these changes Feb 12, 2026
View reviewed changes

github-actionsBot added the approved Indicates a PR has been approved by one committer. label Feb 12, 2026

gavinchou added the dev/4.1.x label Feb 12, 2026

gavinchou merged commit 5cbe1b4 into apache:masterFeb 13, 2026
31 of 32 checks passed

github-actionsBot added the dev/4.0.x-conflict label Feb 13, 2026

Yukang-Lian removed the dev/3.1.x label Mar 30, 2026

Yukang-Lian mentioned this pull request Mar 30, 2026
branch-4.0:[feature](cloud) Support compaction read-write separation for cloud mode #60310 #61872
Closed

Yukang-Lian added a commit to Yukang-Lian/doris that referenced this pull request Mar 30, 2026
[feature](cloud) Support compaction read-write separation for cloud m…
d0ad9d0
…ode (apache#60310)

Yukang-Lian mentioned this pull request Mar 30, 2026
branch-4.1:[feature](cloud) Support compaction read-write separation for cloud mode #60310 #61873
Closed

Yukang-Lian added a commit to Yukang-Lian/doris that referenced this pull request Mar 30, 2026
[feature](cloud) Support compaction read-write separation for cloud m…
e1937f1
…ode (apache#60310)

Yukang-Lian added a commit to Yukang-Lian/doris that referenced this pull request Mar 31, 2026
[feature](cloud) Support compaction read-write separation for cloud m…
2f77523
…ode (apache#60310)

Yukang-Lian added a commit to Yukang-Lian/doris that referenced this pull request Mar 31, 2026
[feature](cloud) Support compaction read-write separation for cloud m…
c03d24b
…ode (apache#60310)

This was referenced Mar 31, 2026
branch-4.0: [pick] Cherry-pick #60520 #60239 #59331 #60310 #61920
Merged
branch-4.1: [pick] Cherry-pick #60520 #60239 #60310 #61921
Merged

yiguolei added dev/4.1.0-merged and removed dev/4.1.x labels Mar 31, 2026

gavinchou added dev/4.0.5-merged and removed dev/4.0.x dev/4.0.x-conflict labels Apr 2, 2026

morningman mentioned this pull request Jun 3, 2026
4.0.6 Release Notes #64065
Closed

morningman mentioned this pull request Jul 3, 2026
[Discuss] Doris Roadmap 2026 #60036
Open
76 tasks

Uh oh!

Conversation

Yukang-Lian commented Jan 28, 2026• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What problem does this PR solve?

Release note

Check List (For Author)

Check List (For Reviewer who merge this PR)

Uh oh!

Thearas commented Jan 28, 2026

Uh oh!

Yukang-Lian commented Jan 29, 2026

Uh oh!

hello-stephen commented Jan 29, 2026

FE UT Coverage Report

Uh oh!

Yukang-Lian commented Jan 29, 2026

Uh oh!

doris-robot commented Jan 29, 2026

Uh oh!

doris-robot commented Jan 29, 2026

Uh oh!

hello-stephen commented Jan 29, 2026

BE UT Coverage Report

Uh oh!

hello-stephen commented Jan 29, 2026

BE Regression && UT Coverage Report

Uh oh!

Yukang-Lian commented Jan 30, 2026

Uh oh!

hello-stephen commented Jan 30, 2026

FE UT Coverage Report

Uh oh!

Uh oh!

Yukang-Lian commented Jan 30, 2026

Uh oh!

hello-stephen commented Jan 30, 2026

FE UT Coverage Report

Uh oh!

doris-robot commented Jan 30, 2026

Uh oh!

doris-robot commented Jan 30, 2026

Uh oh!

doris-robot commented Jan 30, 2026

BE UT Coverage Report

Uh oh!

hello-stephen commented Jan 30, 2026

BE Regression && UT Coverage Report

Uh oh!

Yukang-Lian commented Feb 3, 2026

Uh oh!

doris-robot commented Feb 3, 2026

Cloud UT Coverage Report

Uh oh!

doris-robot commented Feb 3, 2026

Uh oh!

doris-robot commented Feb 3, 2026

Uh oh!

hello-stephen commented Feb 11, 2026

FE Regression Coverage Report

Uh oh!

github-actionsBot commented Feb 12, 2026

Uh oh!

Yukang-Lian commented Feb 12, 2026

Uh oh!

Yukang-Lian commented Feb 12, 2026

Uh oh!

hello-stephen commented Feb 12, 2026

BE Regression && UT Coverage Report

Uh oh!

hello-stephen commented Feb 12, 2026

FE Regression Coverage Report

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Yukang-Lian commented Jan 28, 2026•
edited
Loading