[Fix](StreamingJob) Optimize CDC consumption strategy by JNSimba · Pull Request #60181 · apache/doris

JNSimba · 2026-01-23T09:02:58Z

What problem does this PR solve?

Issue Number: close #xxx

Related PR: #58898 #59461

This PR primarily optimizes the speed of incremental and snapshot reads.

For incremental reads:

Binding the fetch logic to an interval allows fetching data within that interval.
Splitting the fetch and write logic asynchronously.

For snapshot reads:

Introducing the snapshot_split_size and snapshot_parallelism parameters.
snapshot_split_size: Adjusts the size of each chunk during the split phase, allowing each split to fetch more data.
snapshot_parallelism: The degree of parallelism during the snapshot read phase, i.e., how many chunks can run simultaneously, and how many chunks are scheduled in a single task.

Release note

None

Check List (For Author)

Test
- Regression test
- Unit Test
- Manual test (add detailed scripts or steps below)
- No need to test or manual test. Explain why:
  - This is a refactor/code format and no logic has been changed.
  - Previous test can cover this change.
  - No code files have been changed.
  - Other reason
Behavior changed:
- No.
- Yes.
Does this need documentation?
- No.
- Yes.

Check List (For Reviewer who merge this PR)

Confirm the release note
Confirm test cases
Confirm document
Add branch pick label

hello-stephen · 2026-01-23T09:03:04Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

What problem was fixed (it's best to include specific error reporting information). How it was fixed.
Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
What features were added. Why was this function added?
Which code was refactored and why was this part of the code refactored?
Which functions were optimized and what is the difference before and after the optimization?

JNSimba · 2026-01-23T09:05:00Z

run buildall

Copilot

Pull request overview

This PR optimizes the CDC (Change Data Capture) consumption strategy for streaming jobs by refactoring the data polling mechanism and improving heartbeat handling.

Changes:

Refactored the SourceReader interface to split readSplitRecords into two methods: prepareAndSubmitSplit (for split preparation) and pollRecords (for data retrieval)
Moved polling logic from individual readers to the PipelineCoordinator, introducing heartbeat-based synchronization to determine when to stop polling
Reduced the Debezium heartbeat interval from 10 seconds to 3 seconds for faster offset updates and connection issue detection

Reviewed changes

Copilot reviewed 8 out of 8 changed files in this pull request and generated 8 comments.

Show a summary per file

File	Description
Constants.java	Reduced DEBEZIUM_HEARTBEAT_INTERVAL_MS from 10000ms to 3000ms
SourceReader.java	Split readSplitRecords into prepareAndSubmitSplit and pollRecords methods
SplitReadResult.java	Removed recordIterator field as polling is now handled at coordinator level
JdbcIncrementalSourceReader.java	Implemented new split preparation/polling pattern, removed old pollUntilDataAvailable logic
MySqlSourceReader.java	Similar refactoring as JdbcIncrementalSourceReader for MySQL-specific implementation
PostgresSourceReader.java	Added heartbeat interval configuration for Postgres CDC
PipelineCoordinator.java	Major refactoring: added heartbeat-aware polling loops in fetchRecords and writeRecords, extracted helper methods for cleanup and offset extraction
StreamingMultiTblTask.java	Minor log message simplification

Comments suppressed due to low confidence (1)

fs_brokers/cdc_client/src/main/java/org/apache/doris/cdcclient/service/PipelineCoordinator.java:355

The closeJobStreamLoad method is not synchronized, which could lead to a race condition with getOrCreateBatchStreamLoad. If closeJobStreamLoad is called while another thread is accessing the batch stream load, the stream load could be removed and closed while being used, potentially causing NullPointerException or data loss. Consider synchronizing this method or using a more fine-grained locking mechanism.

 public void closeJobStreamLoad(Long jobId) {
DorisBatchStreamLoad batchStreamLoad = batchStreamLoadMap.remove(jobId);
if (batchStreamLoad != null) {
LOG.info("Close DorisBatchStreamLoad for jobId={}", jobId);
batchStreamLoad.close();
batchStreamLoad = null;
}
}

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

doris-robot · 2026-01-23T09:41:11Z

TPC-H: Total hot run time: 30679 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 6bfc454ee6d045f4fc0697234b62203eeb073a80, data reload: false
------ Round 1 ----------------------------------
q1	17596	4798	4843	4798
q2	2043	321	199	199
q3	10230	1304	710	710
q4	10192	790	295	295
q5	7540	2072	1819	1819
q6	197	176	139	139
q7	857	687	587	587
q8	9249	1284	1081	1081
q9	4905	4542	4495	4495
q10	6784	1637	1290	1290
q11	524	286	281	281
q12	336	372	213	213
q13	17776	3830	3069	3069
q14	236	227	214	214
q15	604	522	516	516
q16	650	637	574	574
q17	640	803	472	472
q18	6687	6480	6214	6214
q19	1221	968	636	636
q20	394	332	225	225
q21	2560	1935	1904	1904
q22	1016	989	948	948
Total cold run time: 102237 ms
Total hot run time: 30679 ms
----- Round 2, with runtime_filter_mode=off -----
q1	4750	4702	4683	4683
q2	324	390	322	322
q3	2119	2639	2265	2265
q4	1312	1720	1323	1323
q5	4041	3945	3986	3945
q6	210	174	130	130
q7	1908	1856	1660	1660
q8	2771	2445	2468	2445
q9	7455	7109	7198	7109
q10	2585	2750	2397	2397
q11	617	480	469	469
q12	736	820	643	643
q13	3656	4088	3392	3392
q14	312	331	286	286
q15	624	509	502	502
q16	627	682	644	644
q17	1126	1264	1368	1264
q18	8275	8033	7678	7678
q19	872	815	819	815
q20	1965	2037	1945	1945
q21	4880	4500	4103	4103
q22	1092	1028	974	974
Total cold run time: 52257 ms
Total hot run time: 48994 ms

doris-robot · 2026-01-23T09:51:52Z

TPC-DS: Total hot run time: 172675 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 6bfc454ee6d045f4fc0697234b62203eeb073a80, data reload: false
query5	4439	643	510	510
query6	333	232	209	209
query7	4224	459	259	259
query8	355	264	242	242
query9	8700	2858	2820	2820
query10	463	340	278	278
query11	15151	15122	15029	15029
query12	175	119	117	117
query13	1246	466	337	337
query14	5566	3033	2746	2746
query14_1	2684	2639	2651	2639
query15	201	192	175	175
query16	967	507	473	473
query17	973	677	561	561
query18	2440	440	335	335
query19	191	177	156	156
query20	126	119	118	118
query21	224	147	121	121
query22	3919	4198	3789	3789
query23	16140	15956	15347	15347
query23_1	15526	15471	15464	15464
query24	7078	1530	1164	1164
query24_1	1171	1148	1174	1148
query25	545	455	403	403
query26	1238	273	154	154
query27	2764	433	278	278
query28	4574	2146	2129	2129
query29	791	557	455	455
query30	315	248	210	210
query31	778	621	546	546
query32	88	83	76	76
query33	559	372	310	310
query34	899	895	542	542
query35	745	758	699	699
query36	889	923	802	802
query37	132	96	88	88
query38	2729	2673	2675	2673
query39	779	748	741	741
query39_1	715	741	706	706
query40	222	136	122	122
query41	75	67	67	67
query42	98	93	94	93
query43	475	464	421	421
query44	1308	751	739	739
query45	189	190	181	181
query46	831	950	574	574
query47	1438	1513	1376	1376
query48	330	323	248	248
query49	623	451	364	364
query50	677	265	218	218
query51	3763	3741	3728	3728
query52	93	97	82	82
query53	212	225	168	168
query54	298	270	264	264
query55	84	80	77	77
query56	318	311	310	310
query57	1023	1012	894	894
query58	290	277	273	273
query59	1989	2227	2023	2023
query60	352	341	325	325
query61	193	148	147	147
query62	402	371	312	312
query63	189	157	154	154
query64	4982	1159	860	860
query65	3866	3722	3725	3722
query66	1464	429	320	320
query67	15391	15504	15469	15469
query68	2389	1066	734	734
query69	408	309	286	286
query70	1001	959	871	871
query71	299	290	260	260
query72	5251	3119	3188	3119
query73	595	723	325	325
query74	8694	8808	8500	8500
query75	2275	2315	1894	1894
query76	2268	1051	638	638
query77	356	371	308	308
query78	9747	9914	9124	9124
query79	1385	881	586	586
query80	1288	533	447	447
query81	551	258	232	232
query82	1046	152	122	122
query83	360	270	247	247
query84	295	116	97	97
query85	867	481	452	452
query86	410	290	292	290
query87	2833	2820	2736	2736
query88	3512	2570	2522	2522
query89	309	268	236	236
query90	1951	170	163	163
query91	161	158	133	133
query92	77	70	69	69
query93	1148	1039	638	638
query94	642	322	302	302
query95	604	334	373	334
query96	630	506	225	225
query97	2357	2380	2340	2340
query98	213	205	200	200
query99	612	596	504	504
Total cold run time: 245645 ms
Total hot run time: 172675 ms

doris-robot · 2026-01-23T09:56:52Z

ClickBench: Total hot run time: 26.61 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 6bfc454ee6d045f4fc0697234b62203eeb073a80, data reload: false
query1	0.06	0.05	0.05
query2	0.10	0.05	0.04
query3	0.25	0.08	0.09
query4	1.61	0.12	0.11
query5	0.28	0.26	0.27
query6	1.15	0.65	0.65
query7	0.03	0.03	0.03
query8	0.06	0.04	0.04
query9	0.57	0.50	0.50
query10	0.55	0.54	0.56
query11	0.15	0.10	0.10
query12	0.14	0.11	0.11
query13	0.59	0.58	0.58
query14	0.95	0.97	0.92
query15	0.78	0.78	0.79
query16	0.39	0.39	0.39
query17	1.00	1.02	1.07
query18	0.23	0.22	0.21
query19	1.97	1.84	1.89
query20	0.02	0.01	0.01
query21	15.42	0.25	0.14
query22	5.34	0.05	0.04
query23	16.00	0.29	0.10
query24	1.66	0.52	0.25
query25	0.11	0.06	0.08
query26	0.14	0.13	0.14
query27	0.05	0.08	0.04
query28	3.96	1.06	0.88
query29	12.57	3.91	3.12
query30	0.29	0.14	0.11
query31	2.86	0.63	0.39
query32	3.26	0.56	0.45
query33	3.05	3.02	2.97
query34	15.91	5.11	4.40
query35	4.44	4.42	4.50
query36	0.64	0.50	0.49
query37	0.11	0.06	0.06
query38	0.07	0.05	0.04
query39	0.04	0.03	0.03
query40	0.18	0.15	0.13
query41	0.09	0.03	0.03
query42	0.04	0.03	0.03
query43	0.05	0.04	0.04
Total cold run time: 97.16 s
Total hot run time: 26.61 s

hello-stephen · 2026-01-23T10:21:39Z

FE UT Coverage Report

Increment line coverage 0.00% (0/1) 🎉
Increment coverage report
Complete coverage report

hello-stephen · 2026-01-23T12:03:35Z

FE Regression Coverage Report

Increment line coverage 0.00% (0/1) 🎉
Increment coverage report
Complete coverage report

JNSimba · 2026-01-23T12:45:28Z

run buildall

Copilot

Pull request overview

Copilot reviewed 9 out of 9 changed files in this pull request and generated 15 comments.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

doris-robot · 2026-01-23T13:14:49Z

TPC-H: Total hot run time: 31211 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 56dabdea3282b772d63225c7ffc6092bb2e29b2f, data reload: false
------ Round 1 ----------------------------------
q1	17612	4739	4559	4559
q2	2013	338	201	201
q3	10220	1326	760	760
q4	10220	868	315	315
q5	7921	2135	1842	1842
q6	198	179	148	148
q7	875	736	597	597
q8	9328	1427	1136	1136
q9	5012	4796	4645	4645
q10	7000	1709	1330	1330
q11	536	319	290	290
q12	373	398	235	235
q13	17976	3829	3136	3136
q14	241	256	221	221
q15	600	522	521	521
q16	639	630	585	585
q17	657	731	543	543
q18	6627	6350	6344	6344
q19	1328	963	633	633
q20	383	340	233	233
q21	2745	2236	1955	1955
q22	1056	1025	982	982
Total cold run time: 103560 ms
Total hot run time: 31211 ms
----- Round 2, with runtime_filter_mode=off -----
q1	4764	4674	4698	4674
q2	333	402	325	325
q3	2149	2681	2246	2246
q4	1349	1763	1315	1315
q5	4130	3969	3986	3969
q6	220	176	136	136
q7	1910	1905	2154	1905
q8	2643	2403	2553	2403
q9	7153	7193	7122	7122
q10	2581	2689	2382	2382
q11	547	506	493	493
q12	740	847	687	687
q13	3696	4131	3416	3416
q14	301	305	285	285
q15	561	514	507	507
q16	655	700	652	652
q17	1230	1283	1375	1283
q18	8226	7999	7771	7771
q19	866	829	866	829
q20	1949	2091	2004	2004
q21	4905	4645	4331	4331
q22	1099	1051	943	943
Total cold run time: 52007 ms
Total hot run time: 49678 ms

doris-robot · 2026-01-23T13:25:31Z

TPC-DS: Total hot run time: 172749 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 56dabdea3282b772d63225c7ffc6092bb2e29b2f, data reload: false
query5	4463	641	488	488
query6	341	217	192	192
query7	4215	454	276	276
query8	339	242	225	225
query9	8684	2822	2887	2822
query10	430	310	274	274
query11	15444	15278	14937	14937
query12	186	116	116	116
query13	1250	469	372	372
query14	5834	3023	2859	2859
query14_1	2805	2814	2752	2752
query15	202	192	169	169
query16	977	482	480	480
query17	1059	643	561	561
query18	2410	414	319	319
query19	190	175	152	152
query20	121	112	113	112
query21	214	135	113	113
query22	3886	4035	3846	3846
query23	15967	15452	15211	15211
query23_1	15259	15421	15523	15421
query24	7151	1536	1160	1160
query24_1	1170	1165	1189	1165
query25	508	425	382	382
query26	1237	271	157	157
query27	2761	450	278	278
query28	4595	2164	2132	2132
query29	736	539	437	437
query30	309	252	202	202
query31	790	631	538	538
query32	84	75	71	71
query33	516	356	310	310
query34	919	864	536	536
query35	723	753	675	675
query36	878	898	734	734
query37	136	107	93	93
query38	2752	2739	2649	2649
query39	788	745	718	718
query39_1	708	722	722	722
query40	220	139	121	121
query41	74	101	62	62
query42	94	92	90	90
query43	428	474	400	400
query44	1329	746	747	746
query45	187	189	176	176
query46	827	942	596	596
query47	1376	1458	1389	1389
query48	323	322	253	253
query49	638	435	344	344
query50	684	269	201	201
query51	3793	3749	3759	3749
query52	93	97	84	84
query53	207	220	170	170
query54	282	252	245	245
query55	77	80	77	77
query56	297	298	295	295
query57	1002	971	856	856
query58	269	260	256	256
query59	2106	2128	1935	1935
query60	334	323	310	310
query61	147	143	141	141
query62	367	375	331	331
query63	198	165	164	164
query64	4880	1144	826	826
query65	3836	3769	3791	3769
query66	1468	441	332	332
query67	15553	15577	15494	15494
query68	2479	1061	721	721
query69	427	331	294	294
query70	924	986	933	933
query71	307	294	272	272
query72	5456	3184	3356	3184
query73	592	726	317	317
query74	8728	8780	8588	8588
query75	2352	2334	1979	1979
query76	2244	1049	661	661
query77	350	382	311	311
query78	9793	9847	9150	9150
query79	1966	918	601	601
query80	1499	503	445	445
query81	539	269	240	240
query82	1005	147	118	118
query83	327	258	239	239
query84	252	123	95	95
query85	876	469	400	400
query86	409	297	280	280
query87	2900	2873	2835	2835
query88	3533	2596	2565	2565
query89	307	270	237	237
query90	1984	183	165	165
query91	160	154	130	130
query92	71	72	74	72
query93	1219	1021	645	645
query94	632	324	279	279
query95	573	336	315	315
query96	643	509	240	240
query97	2344	2365	2305	2305
query98	234	197	205	197
query99	608	561	515	515
Total cold run time: 247066 ms
Total hot run time: 172749 ms

doris-robot · 2026-01-23T13:30:32Z

ClickBench: Total hot run time: 27.19 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 56dabdea3282b772d63225c7ffc6092bb2e29b2f, data reload: false
query1	0.05	0.05	0.04
query2	0.10	0.05	0.05
query3	0.26	0.09	0.08
query4	1.61	0.12	0.12
query5	0.28	0.25	0.25
query6	1.14	0.65	0.65
query7	0.02	0.02	0.02
query8	0.05	0.04	0.04
query9	0.57	0.50	0.49
query10	0.55	0.56	0.54
query11	0.14	0.10	0.10
query12	0.14	0.11	0.11
query13	0.61	0.58	0.58
query14	0.96	0.95	0.94
query15	0.80	0.78	0.80
query16	0.40	0.40	0.41
query17	0.99	0.98	1.05
query18	0.23	0.22	0.21
query19	1.86	1.90	1.83
query20	0.02	0.01	0.01
query21	15.43	0.27	0.14
query22	5.04	0.05	0.05
query23	15.93	0.29	0.10
query24	0.92	0.70	0.59
query25	0.11	0.15	0.13
query26	0.14	0.13	0.13
query27	0.08	0.05	0.06
query28	5.03	1.07	0.88
query29	12.53	4.06	3.22
query30	0.28	0.14	0.14
query31	2.82	0.62	0.39
query32	3.23	0.56	0.46
query33	2.96	2.96	3.06
query34	16.12	5.09	4.41
query35	4.47	4.50	4.49
query36	0.66	0.51	0.49
query37	0.11	0.07	0.06
query38	0.07	0.05	0.04
query39	0.04	0.02	0.03
query40	0.16	0.14	0.13
query41	0.08	0.04	0.03
query42	0.04	0.03	0.02
query43	0.05	0.04	0.04
Total cold run time: 97.08 s
Total hot run time: 27.19 s

hello-stephen · 2026-01-23T15:48:14Z

FE Regression Coverage Report

Increment line coverage 0.00% (0/1) 🎉
Increment coverage report
Complete coverage report

JNSimba · 2026-01-26T03:30:43Z

run buildall

JNSimba · 2026-01-26T03:44:48Z

run buildall

doris-robot · 2026-01-26T05:18:09Z

TPC-H: Total hot run time: 31339 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 1e727935827a2fefde5f0b9512e5b71603c80744, data reload: false
------ Round 1 ----------------------------------
q1	17720	4821	4574	4574
q2	2028	336	204	204
q3	10225	1325	727	727
q4	10194	767	307	307
q5	7512	2122	1846	1846
q6	199	169	139	139
q7	887	747	583	583
q8	9290	1396	1143	1143
q9	4972	4565	4506	4506
q10	6817	1665	1277	1277
q11	538	291	277	277
q12	396	374	219	219
q13	17791	3863	3074	3074
q14	245	248	219	219
q15	609	520	537	520
q16	639	629	582	582
q17	674	858	454	454
q18	6747	6559	6822	6559
q19	1466	1036	658	658
q20	410	374	248	248
q21	3065	2275	2150	2150
q22	1130	1073	1093	1073
Total cold run time: 103554 ms
Total hot run time: 31339 ms
----- Round 2, with runtime_filter_mode=off -----
q1	5090	4910	4945	4910
q2	335	411	347	347
q3	2405	2805	2411	2411
q4	1494	1878	1437	1437
q5	4546	4489	4397	4397
q6	220	177	137	137
q7	2054	2009	1800	1800
q8	2590	2396	2372	2372
q9	7167	7314	7070	7070
q10	2455	2574	2091	2091
q11	537	463	450	450
q12	674	703	552	552
q13	3298	3818	3032	3032
q14	282	294	262	262
q15	535	500	487	487
q16	607	662	609	609
q17	1097	1327	1288	1288
q18	7468	7349	7153	7153
q19	856	774	815	774
q20	1881	1942	1813	1813
q21	4530	4277	4083	4083
q22	1066	1018	966	966
Total cold run time: 51187 ms
Total hot run time: 48441 ms

doris-robot · 2026-01-26T05:28:50Z

TPC-DS: Total hot run time: 172273 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 1e727935827a2fefde5f0b9512e5b71603c80744, data reload: false
query5	4431	624	481	481
query6	325	211	196	196
query7	4214	451	262	262
query8	336	229	226	226
query9	8687	2844	2857	2844
query10	474	337	273	273
query11	15405	15182	14819	14819
query12	175	122	118	118
query13	1255	467	368	368
query14	6161	3000	2795	2795
query14_1	2634	2614	2652	2614
query15	205	194	173	173
query16	984	486	463	463
query17	1092	673	578	578
query18	2485	426	333	333
query19	190	184	168	168
query20	122	114	119	114
query21	217	135	119	119
query22	4218	4132	4265	4132
query23	15949	15513	15158	15158
query23_1	15401	15421	15351	15351
query24	7065	1553	1187	1187
query24_1	1170	1164	1138	1138
query25	589	460	407	407
query26	1241	261	159	159
query27	2763	462	280	280
query28	4553	2193	2188	2188
query29	788	531	449	449
query30	309	230	206	206
query31	799	647	558	558
query32	83	76	76	76
query33	532	345	339	339
query34	940	863	536	536
query35	703	752	681	681
query36	891	873	837	837
query37	132	96	87	87
query38	2756	2692	2636	2636
query39	777	737	746	737
query39_1	734	715	705	705
query40	229	137	120	120
query41	73	70	66	66
query42	98	94	94	94
query43	476	462	430	430
query44	1316	759	747	747
query45	189	189	181	181
query46	836	948	618	618
query47	1483	1495	1351	1351
query48	324	329	245	245
query49	602	448	351	351
query50	668	265	208	208
query51	3744	3866	3701	3701
query52	97	89	80	80
query53	207	223	162	162
query54	273	252	239	239
query55	80	80	73	73
query56	283	285	286	285
query57	1056	1004	906	906
query58	260	253	260	253
query59	2107	2140	2189	2140
query60	346	327	311	311
query61	145	143	139	139
query62	390	367	313	313
query63	192	161	155	155
query64	4866	1141	825	825
query65	3740	3727	3757	3727
query66	1439	409	326	326
query67	15621	15701	15443	15443
query68	2399	1049	713	713
query69	393	307	278	278
query70	990	898	868	868
query71	300	293	254	254
query72	5261	3124	3203	3124
query73	617	720	305	305
query74	8689	8675	8458	8458
query75	2297	2319	1877	1877
query76	2267	1027	648	648
query77	353	367	300	300
query78	9823	9964	9224	9224
query79	1126	913	577	577
query80	1314	501	431	431
query81	547	267	235	235
query82	1266	147	118	118
query83	344	255	236	236
query84	253	115	94	94
query85	1006	478	393	393
query86	416	293	287	287
query87	2843	2850	2716	2716
query88	3536	2610	2581	2581
query89	302	267	237	237
query90	1917	170	172	170
query91	164	158	132	132
query92	82	74	71	71
query93	1093	1013	641	641
query94	640	310	288	288
query95	583	385	307	307
query96	640	512	227	227
query97	2348	2364	2324	2324
query98	213	204	198	198
query99	579	578	510	510
Total cold run time: 246739 ms
Total hot run time: 172273 ms

doris-robot · 2026-01-26T05:33:50Z

ClickBench: Total hot run time: 26.65 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 1e727935827a2fefde5f0b9512e5b71603c80744, data reload: false
query1	0.05	0.05	0.05
query2	0.09	0.05	0.04
query3	0.25	0.09	0.08
query4	1.61	0.11	0.11
query5	0.27	0.26	0.25
query6	1.15	0.65	0.65
query7	0.03	0.02	0.02
query8	0.05	0.04	0.04
query9	0.57	0.50	0.49
query10	0.54	0.53	0.54
query11	0.14	0.09	0.09
query12	0.14	0.11	0.11
query13	0.60	0.59	0.59
query14	0.95	0.93	0.94
query15	0.79	0.78	0.79
query16	0.38	0.40	0.39
query17	1.07	1.06	1.02
query18	0.23	0.22	0.21
query19	2.00	1.73	1.92
query20	0.02	0.01	0.01
query21	15.44	0.26	0.13
query22	5.06	0.06	0.05
query23	15.86	0.28	0.10
query24	1.75	0.29	0.52
query25	0.10	0.06	0.05
query26	0.16	0.13	0.13
query27	0.08	0.08	0.05
query28	4.30	1.07	0.88
query29	12.51	3.91	3.15
query30	0.28	0.14	0.12
query31	2.81	0.62	0.39
query32	3.24	0.57	0.46
query33	3.04	3.08	3.04
query34	15.83	4.99	4.41
query35	4.42	4.44	4.45
query36	0.65	0.50	0.49
query37	0.11	0.07	0.06
query38	0.07	0.04	0.04
query39	0.04	0.03	0.03
query40	0.16	0.14	0.13
query41	0.09	0.04	0.03
query42	0.04	0.02	0.03
query43	0.05	0.04	0.04
Total cold run time: 97.02 s
Total hot run time: 26.65 s

JNSimba · 2026-02-02T15:16:29Z

run cloud_p0

hello-stephen · 2026-02-02T15:37:12Z

FE Regression Coverage Report

Increment line coverage 76.27% (90/118) 🎉
Increment coverage report
Complete coverage report

hello-stephen · 2026-02-02T15:52:28Z

FE Regression Coverage Report

Increment line coverage 76.27% (90/118) 🎉
Increment coverage report
Complete coverage report

JNSimba · 2026-02-03T00:07:03Z

run cloud_p0

hello-stephen · 2026-02-03T01:07:10Z

FE Regression Coverage Report

Increment line coverage 76.27% (90/118) 🎉
Increment coverage report
Complete coverage report

JNSimba · 2026-02-03T01:59:55Z

run cloud_p0

JNSimba · 2026-02-03T02:42:39Z

run buildall

hello-stephen · 2026-02-03T02:57:07Z

FE Regression Coverage Report

Increment line coverage 76.27% (90/118) 🎉
Increment coverage report
Complete coverage report

JNSimba · 2026-02-03T03:04:34Z

run buildall

JNSimba · 2026-02-03T03:31:36Z

run buildall

doris-robot · 2026-02-03T04:02:06Z

TPC-H: Total hot run time: 31939 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit b5a25f146c54eeab1aa8ea9b8fb0bd04db3df821, data reload: false
------ Round 1 ----------------------------------
q1	17631	5218	5066	5066
q2	2062	321	211	211
q3	10178	1286	739	739
q4	10212	878	308	308
q5	7545	2211	1870	1870
q6	192	178	147	147
q7	918	730	604	604
q8	9267	1429	1095	1095
q9	5191	4857	4849	4849
q10	6783	1973	1582	1582
q11	487	302	276	276
q12	333	372	223	223
q13	17756	4073	3250	3250
q14	236	255	216	216
q15	895	811	828	811
q16	687	701	630	630
q17	657	771	511	511
q18	6696	6539	6528	6528
q19	1236	992	639	639
q20	409	358	231	231
q21	2673	2084	1873	1873
q22	365	319	280	280
Total cold run time: 102409 ms
Total hot run time: 31939 ms
----- Round 2, with runtime_filter_mode=off -----
q1	5315	5338	5302	5302
q2	261	345	249	249
q3	2190	2660	2247	2247
q4	1365	1732	1279	1279
q5	4254	4163	4198	4163
q6	218	181	136	136
q7	2218	2176	1839	1839
q8	2627	2403	2403	2403
q9	7558	7467	7583	7467
q10	2785	3021	2663	2663
q11	594	474	481	474
q12	702	779	649	649
q13	3918	4453	3532	3532
q14	300	393	309	309
q15	881	838	833	833
q16	719	701	684	684
q17	1235	1400	1345	1345
q18	7984	8116	7862	7862
q19	953	879	895	879
q20	2043	2192	1964	1964
q21	4754	4446	4385	4385
q22	583	538	549	538
Total cold run time: 53457 ms
Total hot run time: 51202 ms

doris-robot · 2026-02-03T04:18:55Z

ClickBench: Total hot run time: 28.35 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit b5a25f146c54eeab1aa8ea9b8fb0bd04db3df821, data reload: false
query1	0.06	0.04	0.04
query2	0.09	0.04	0.04
query3	0.26	0.09	0.09
query4	1.60	0.12	0.11
query5	0.26	0.24	0.25
query6	1.16	0.67	0.67
query7	0.03	0.02	0.02
query8	0.06	0.04	0.04
query9	0.56	0.51	0.49
query10	0.56	0.54	0.55
query11	0.13	0.10	0.10
query12	0.14	0.10	0.11
query13	0.63	0.61	0.62
query14	1.08	1.04	1.05
query15	0.87	0.86	0.87
query16	0.39	0.40	0.42
query17	1.09	1.15	1.15
query18	0.23	0.21	0.21
query19	2.06	2.01	1.94
query20	0.02	0.02	0.01
query21	15.41	0.26	0.16
query22	5.00	0.05	0.04
query23	15.78	0.29	0.11
query24	1.60	0.34	1.30
query25	0.12	0.08	0.06
query26	0.14	0.14	0.14
query27	0.08	0.07	0.05
query28	5.15	1.15	0.99
query29	12.54	3.90	3.14
query30	0.27	0.13	0.11
query31	2.81	0.63	0.39
query32	3.24	0.60	0.50
query33	3.28	3.22	3.28
query34	16.19	5.38	4.76
query35	4.80	4.85	4.76
query36	0.65	0.49	0.49
query37	0.11	0.07	0.07
query38	0.07	0.04	0.04
query39	0.04	0.04	0.03
query40	0.19	0.16	0.16
query41	0.09	0.03	0.03
query42	0.04	0.03	0.03
query43	0.05	0.04	0.03
Total cold run time: 98.93 s
Total hot run time: 28.35 s

github-actions · 2026-02-03T06:10:58Z

PR approved by at least one committer and no changes requested.

### What problem does this PR solve? Related PR: #58898 #59461 This PR primarily optimizes the speed of incremental and snapshot reads. 1. For incremental reads: - Binding the fetch logic to an interval allows fetching data within that interval. - Splitting the fetch and write logic asynchronously. 2. For snapshot reads: - Introducing the `snapshot_split_size` and `snapshot_parallelism` parameters. - `snapshot_split_size`: Adjusts the size of each chunk during the split phase, allowing each split to fetch more data. - `snapshot_parallelism`: The degree of parallelism during the snapshot read phase, i.e., how many chunks can run simultaneously, and how many chunks are scheduled in a single task.

hello-stephen · 2026-02-03T06:27:24Z

FE Regression Coverage Report

Increment line coverage 1.69% (2/118) 🎉
Increment coverage report
Complete coverage report

(#60463) Cherry-picked from #60181 Co-authored-by: wudi <wudi@selectdb.com>

…che#60181 (apache#60463) Cherry-picked from apache#60181 Co-authored-by: wudi <wudi@selectdb.com>

1. change produce -consumer to poll split record，poll until headerbeat
6bfc454

JNSimba requested a review from Copilot January 23, 2026 09:05

Copilotstarted reviewing on behalf of JNSimba January 23, 2026 09:05 View session

CopilotAI reviewed Jan 23, 2026
View reviewed changes

fix
56dabde

JNSimba requested a review from Copilot January 23, 2026 12:45

Copilotstarted reviewing on behalf of JNSimba January 23, 2026 12:45 View session

CopilotAI reviewed Jan 23, 2026
View reviewed changes

add log
56f63e3

JNSimba requested review from CalvinKirs and morningman as code owners January 26, 2026 03:26

fix
1e72793

fix
dbd5a0f

JNSimba added 2 commits February 3, 2026 10:39

fix regress conf
df540b4

fix
f29a1bd

JNSimba added 2 commits February 3, 2026 10:59

code style
bb73aab

fix pom
4853a09

fix
b5a25f1

liaoxin01 approved these changes Feb 3, 2026
View reviewed changes

github-actionsBot added the approved Indicates a PR has been approved by one committer. label Feb 3, 2026

CalvinKirs approved these changes Feb 3, 2026
View reviewed changes

JNSimba merged commit ae5ff35 into apache:masterFeb 3, 2026
29 of 30 checks passed

JNSimba added the dev/4.0.x label Feb 3, 2026

github-actionsBot mentioned this pull request Feb 3, 2026
branch-4.0: [Fix](StreamingJob) Optimize CDC consumption strategy #60181 #60463
Merged

yiguolei added dev/4.0.4-merged and removed dev/4.0.x labels Feb 4, 2026

yiguolei pushed a commit that referenced this pull request Feb 4, 2026
branch-4.0: [Fix](StreamingJob) Optimize CDC consumption strategy#60181
691a881
(#60463) Cherry-picked from #60181 Co-authored-by: wudi <wudi@selectdb.com>

ybtsdst pushed a commit to ybtsdst/doris that referenced this pull request Feb 27, 2026
branch-4.0: [Fix](StreamingJob) Optimize CDC consumption strategyapa…
e6c6cc2
…che#60181 (apache#60463) Cherry-picked from apache#60181 Co-authored-by: wudi <wudi@selectdb.com>

Uh oh!

Conversation

JNSimba commented Jan 23, 2026• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What problem does this PR solve?

Release note

Check List (For Author)

Check List (For Reviewer who merge this PR)

Uh oh!

hello-stephen commented Jan 23, 2026

Uh oh!

JNSimba commented Jan 23, 2026

Uh oh!

CopilotAI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

doris-robot commented Jan 23, 2026

Uh oh!

doris-robot commented Jan 23, 2026

Uh oh!

doris-robot commented Jan 23, 2026

Uh oh!

hello-stephen commented Jan 23, 2026

FE UT Coverage Report

Uh oh!

hello-stephen commented Jan 23, 2026

FE Regression Coverage Report

Uh oh!

JNSimba commented Jan 23, 2026

Uh oh!

CopilotAI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

doris-robot commented Jan 23, 2026

Uh oh!

doris-robot commented Jan 23, 2026

Uh oh!

doris-robot commented Jan 23, 2026

Uh oh!

hello-stephen commented Jan 23, 2026

FE Regression Coverage Report

Uh oh!

JNSimba commented Jan 26, 2026

Uh oh!

JNSimba commented Jan 26, 2026

Uh oh!

doris-robot commented Jan 26, 2026

Uh oh!

doris-robot commented Jan 26, 2026

Uh oh!

doris-robot commented Jan 26, 2026

Uh oh!

JNSimba commented Feb 2, 2026

Uh oh!

hello-stephen commented Feb 2, 2026

FE Regression Coverage Report

Uh oh!

hello-stephen commented Feb 2, 2026

FE Regression Coverage Report

JNSimba commented Jan 23, 2026•
edited
Loading