[improve](streaming-job) support user-specified mysql server_id with per-reader assignment by JNSimba · Pull Request #63490 · apache/doris

JNSimba · 2026-05-21T11:00:31Z

Summary

Add an optional server_id source property for MySQL CDC streaming jobs. Accepts a single value (e.g. 5400) or a range (e.g. 5400-5408). When unset, the value is derived from the jobId hash so existing jobs keep their current server_id when snapshot_parallelism = 1.
Fix a latent collision: when snapshot_parallelism > 1 and source-side DML happens during snapshot, all parallel SnapshotSplitReader instances previously shared the same server_id and their backfill BinaryLogClient connections kicked each other out of MySQL's dump-thread slot, dropping binlog events between low and high watermark. Each subtask now gets a distinct server_id from the resolved range; the single binlog reader uses the range start.
Cross-field check: reject server_id range width smaller than snapshot_parallelism at job startup with a clear fix-it suggestion.

Test plan

ConfigUtilTest: 15 cases covering default-derive (Integer.MAX_VALUE hash clamp, hash=0 bump), user single value / range, malformed input, blank input, range-vs-parallelism width check, non-positive parallelism.
test_streaming_mysql_job_server_id.groovy: 4 rejection cases (format / zero / backward / width) under offset=latest for synchronous CREATE JOB feedback, plus 3 happy-path cases verifying snapshot data syncs under single value / range / default-derive.

hello-stephen · 2026-05-21T11:00:38Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

What problem was fixed (it's best to include specific error reporting information). How it was fixed.
Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
What features were added. Why was this function added?
Which code was refactored and why was this part of the code refactored?
Which functions were optimized and what is the difference before and after the optimization?

JNSimba · 2026-05-21T11:01:22Z

/review

JNSimba · 2026-05-21T11:01:31Z

run buildall

Copilot

Pull request overview

Adds support for a user-configurable MySQL CDC server_id (single value or range) for streaming jobs, and ensures parallel snapshot readers use distinct server_ids to avoid MySQL dump-thread collisions during snapshot backfill.

Changes:

Introduces server_id as a supported MySQL source property (and allows it through FE validation).
Adds ConfigUtil.resolveServerIdRange() to parse/validate user input or deterministically derive a range from jobId sized to snapshot_parallelism.
Updates MySqlSourceReader to build per-subtask MySQL source configs so each snapshot reader binds a unique server_id, plus adds unit/regression coverage.

Reviewed changes

Copilot reviewed 6 out of 6 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
regression-test/suites/job_p0/streaming_job/cdc/test_streaming_mysql_job_server_id.groovy	Adds regression coverage for rejecting invalid `server_id` inputs and basic happy-path sync checks.
fs_brokers/cdc_client/src/test/java/org/apache/doris/cdcclient/utils/ConfigUtilTest.java	Adds unit tests for default-derived and user-specified `server_id` range resolution/validation.
fs_brokers/cdc_client/src/main/java/org/apache/doris/cdcclient/utils/ConfigUtil.java	Implements `server_id` range resolution, derivation, and cross-field validation vs `snapshot_parallelism`.
fs_brokers/cdc_client/src/main/java/org/apache/doris/cdcclient/source/reader/mysql/MySqlSourceReader.java	Applies per-reader `server_id` assignment by generating per-subtask configs from a resolved range.
fe/fe-core/src/main/java/org/apache/doris/job/extensions/insert/streaming/DataSourceConfigValidator.java	Allows `server_id` in streaming job source properties.
fe/fe-common/src/main/java/org/apache/doris/job/cdc/DataSourceConfigKeys.java	Adds the `server_id` config key and documents expected formats.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

github-actions

I found one blocking issue in the server_id validation path. The PR's goal is clear and the reader-side per-subtask assignment is focused, but CREATE/ALTER validation does not currently enforce the new property's contract, so bad values can be accepted and persisted before the CDC client later fails.

Critical checkpoints: goal/test coverage: the feature is covered by unit/regression tests, but the new regression expectation for synchronous CREATE rejection is not satisfied by the current initReader path. Scope: the implementation is small and localized. Concurrency/lifecycle: no new shared mutable lifecycle issue found in the reader assignment path. Config items: server_id is a new source property and needs FE-side validation because it is user-facing and persisted. Compatibility/persistence: no storage-format or FE-BE protocol compatibility issue found. Parallel paths: CREATE/ALTER source validation should both reject malformed server_id consistently. Observability/performance/transactions: no additional issue found. User focus: no additional user-provided review focus.

hello-stephen · 2026-05-21T12:02:05Z

TPC-H: Total hot run time: 30804 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 5696cc20023456eb0d5eddbf100a80cc2c60e531, data reload: false
------ Round 1 ----------------------------------
orders	Doris	NULL	NULL	0	0	0	NULL	0	NULL	NULL	2023-12-26 18:27:23	2023-12-26 18:42:55	NULL	utf-8	NULL	NULL ============================================
q1	17640	3859	3869	3859
q2	q3	10793	1436	822	822
q4	4696	490	356	356
q5	7667	2272	2112	2112
q6	373	178	138	138
q7	1006	763	643	643
q8	9364	1701	1473	1473
q9	6653	4961	4905	4905
q10	6440	2136	1808	1808
q11	441	277	248	248
q12	695	422	296	296
q13	18307	3456	2821	2821
q14	261	252	240	240
q15	q16	818	762	701	701
q17	917	1002	969	969
q18	6876	5642	5422	5422
q19	1161	1294	1030	1030
q20	511	407	256	256
q21	5694	2661	2402	2402
q22	434	354	303	303
Total cold run time: 100747 ms
Total hot run time: 30804 ms
----- Round 2, with runtime_filter_mode=off -----
orders	Doris	NULL	NULL	150000000	42	6422171781	NULL	22778155	NULL	NULL	2023-12-26 18:27:23	2023-12-26 18:42:55	NULL	utf-8	NULL	NULL ============================================
q1	4165	4144	4082	4082
q2	q3	4510	4900	4365	4365
q4	2119	2236	1410	1410
q5	4391	4309	4278	4278
q6	229	170	129	129
q7	2181	1920	1666	1666
q8	2525	2133	2062	2062
q9	7801	7802	7815	7802
q10	4507	4457	4100	4100
q11	569	465	383	383
q12	730	737	588	588
q13	3404	3671	3091	3091
q14	296	340	292	292
q15	q16	732	746	703	703
q17	1356	1305	1310	1305
q18	8229	7458	7051	7051
q19	1134	1087	1133	1087
q20	2217	2219	1934	1934
q21	5352	4870	4521	4521
q22	530	495	417	417
Total cold run time: 56977 ms
Total hot run time: 51266 ms

hello-stephen · 2026-05-21T12:12:59Z

TPC-DS: Total hot run time: 169540 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 5696cc20023456eb0d5eddbf100a80cc2c60e531, data reload: false
query5	4316	669	528	528
query6	355	222	201	201
query7	4280	570	303	303
query8	331	239	226	226
query9	8826	4045	4045	4045
query10	468	373	301	301
query11	5825	2410	2223	2223
query12	172	129	127	127
query13	1290	612	450	450
query14	5996	5377	5073	5073
query14_1	4523	4394	4369	4369
query15	209	204	187	187
query16	1029	467	433	433
query17	1162	735	606	606
query18	2741	512	360	360
query19	223	209	182	182
query20	142	131	133	131
query21	219	138	117	117
query22	13624	13662	13447	13447
query23	17291	16382	15967	15967
query23_1	16069	16193	16179	16179
query24	7458	1747	1297	1297
query24_1	1343	1292	1296	1292
query25	595	489	438	438
query26	1303	325	181	181
query27	2687	571	339	339
query28	4435	1977	1980	1977
query29	1016	643	516	516
query30	308	242	203	203
query31	1119	1072	946	946
query32	97	78	81	78
query33	552	363	314	314
query34	1176	1101	650	650
query35	774	823	671	671
query36	1292	1290	1209	1209
query37	149	103	89	89
query38	3216	3147	3069	3069
query39	937	917	904	904
query39_1	865	872	877	872
query40	233	148	122	122
query41	67	65	63	63
query42	109	112	109	109
query43	323	327	285	285
query44 query45	214	200	193	193
query46	1068	1199	701	701
query47	2284	2315	2112	2112
query48	405	408	304	304
query49	628	498	381	381
query50	941	335	263	263
query51	4319	4323	4240	4240
query52	107	106	94	94
query53	250	282	203	203
query54	315	270	259	259
query55	95	90	86	86
query56	308	327	304	304
query57	1392	1376	1293	1293
query58	307	278	275	275
query59	1544	1641	1399	1399
query60	334	323	315	315
query61	159	156	158	156
query62	679	629	549	549
query63	244	198	206	198
query64	2405	794	626	626
query65 query66	1662	482	352	352
query67	29681	29973	29965	29965
query68 query69	469	340	295	295
query70	1034	942	984	942
query71	310	278	276	276
query72	3060	2733	2369	2369
query73	856	802	429	429
query74	5050	4992	4709	4709
query75	2677	2627	2259	2259
query76	2297	1129	751	751
query77	395	401	328	328
query78	12232	12277	11589	11589
query79	1475	1026	760	760
query80	914	533	448	448
query81	503	276	236	236
query82	1364	159	127	127
query83	358	292	265	265
query84	256	145	109	109
query85	938	539	462	462
query86	452	355	303	303
query87	3473	3361	3243	3243
query88	3529	2687	2682	2682
query89	442	381	334	334
query90	1780	187	189	187
query91	183	169	142	142
query92	81	77	72	72
query93	1471	1450	835	835
query94	624	354	305	305
query95	682	471	358	358
query96	1014	824	368	368
query97	2693	2723	2585	2585
query98	237	243	250	243
query99	1095	1104	1022	1022
Total cold run time: 253400 ms
Total hot run time: 169540 ms

hello-stephen · 2026-05-21T12:19:48Z

FE UT Coverage Report

Increment line coverage 100% (0/0) 🎉
Increment coverage report
Complete coverage report

hello-stephen · 2026-05-21T14:10:07Z

FE Regression Coverage Report

Increment line coverage 100% (0/0) 🎉
Increment coverage report
Complete coverage report

JNSimba · 2026-05-22T03:00:07Z

run buildall

JNSimba · 2026-05-22T03:00:12Z

/review

github-actions

I completed the full PR review and did not find additional distinct inline issues beyond the already-open review context. I am still requesting changes because the existing FE validation thread remains a blocking correctness gap: server_id is accepted by DataSourceConfigValidator without source-type-specific syntax/range validation, so initial/snapshot CREATE JOB paths can persist invalid values before the CDC reader parses them. I am not adding a duplicate inline comment for that already-known thread.

Critical checkpoint conclusions:

Goal/test: The PR aims to let MySQL CDC jobs use a user-provided or derived server_id range and adds unit/regression coverage. The reader-side range resolution and per-subtask config accomplish part of this, but CREATE-time validation is incomplete for initial/snapshot jobs.
Scope/focus: The implementation is mostly small and focused.
Concurrency: The new per-reader IDs align with the existing snapshot polling concurrency model; I did not find a new distinct concurrency issue.
Lifecycle/static init: No special static/global lifecycle issue found.
Configuration: server_id is a new source config; it needs FE validation at the persisted job boundary, especially because not all create paths initialize the reader.
Compatibility/persistence: Persisting invalid config is the main compatibility/correctness risk; no storage-format change found.
Parallel paths: MySQL reader paths were updated; FE validation remains the uncovered parallel boundary.
Conditions/error handling: Reader-side errors are explicit, but they occur too late for initial/snapshot create paths.
Tests: Unit tests cover resolver behavior and regression tests cover happy paths plus latest-mode rejection, but they do not prove invalid initial/snapshot server_id is rejected synchronously.
Observability/performance/transactions: No additional issue found.

User focus: No additional user-provided review focus was specified.

JNSimba · 2026-05-22T07:46:44Z

/review

github-actions

I found one blocking issue in the new server_id validation path.

Critical checkpoint conclusions:

Goal/test coverage: the PR adds MySQL CDC server_id support and tests for create-time validation, but ALTER JOB coverage misses merged-property validation.
Scope/clarity: the change is mostly focused, but the cross-field validator is called from an ALTER path with incomplete input.
Concurrency/lifecycle: no new shared-state concurrency issue found in the reader changes; the issue is configuration lifecycle across paused ALTER and later task execution.
Configuration compatibility: server_id is persisted job configuration; ALTER must validate the effective persisted config before accepting it.
Parallel paths: CREATE validates the full source map, but ALTER validates only the source-property delta, so the paths are inconsistent.
Tests: new CREATE/regression coverage does not cover ALTERing server_id on an existing job with snapshot_parallelism > 1.
Focus points: no additional user-provided focus points were supplied.

Please fix the ALTER path to validate the merged source properties, or make server_id immutable when snapshot_parallelism is already fixed.

JNSimba · 2026-05-25T08:24:53Z

/review

github-actions

Review conclusion: I did not find additional non-duplicate inline findings beyond the existing review threads, but the current head still needs changes before merge. The already-open threads around FE server_id validation/source-type handling and ALTER JOB effective-property validation remain the blocking items to resolve.

Critical checkpoints:

Goal/test coverage: the PR aims to let MySQL CDC jobs configure safe server_id ranges and to derive distinct IDs for parallel snapshot readers. The cdc_client path now uses ranges and there are FE/cdc_client unit tests plus a MySQL regression test, but the existing review threads identify CREATE/ALTER validation gaps that mean the tests do not fully prove invalid persisted/effective configs are rejected at analysis time.
Scope/focus: the implementation is mostly focused on server_id handling, with small changes in FE validation, TVF validation, cdc_client config generation, and tests.
Concurrency: the runtime change affects parallel snapshot readers by assigning per-subtask server IDs. I did not find a new lock-order or shared-state concurrency issue in this pass.
Lifecycle/static initialization: no new static initialization-order or lifecycle-management issue found.
Configuration compatibility: server_id is a new source property and existing jobs without it continue to derive IDs, but the already-open validation comments need resolution for invalid or non-MySQL usage and ALTER merged-property handling.
Parallel code paths: cdc_stream TVF and CREATE JOB paths were considered; no additional distinct path issue found beyond the existing validation threads.
Error handling: no ignored Status-style issue applies in this Java path; user-facing validation errors are generally propagated as analysis failures, subject to the existing gaps.
Persistence/transactions/data correctness: no new transaction or persistence issue found; the concern remains preventing invalid source configs from being persisted/effective.
Observability/performance: no additional blocking issue found; added server_id logs are lightweight.
User focus: no additional user-provided review focus was specified.

Please address the existing inline threads before this PR is merged.

JNSimba · 2026-05-26T03:46:58Z

run buildall

hello-stephen · 2026-05-26T05:10:18Z

FE UT Coverage Report

Increment line coverage 74.47% (35/47) 🎉
Increment coverage report
Complete coverage report

hello-stephen · 2026-05-26T06:04:31Z

TPC-H: Total hot run time: 31215 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 918be4b864583885cb43290315b60963efa4f209, data reload: false
------ Round 1 ----------------------------------
orders	Doris	NULL	NULL	0	0	0	NULL	0	NULL	NULL	2023-12-26 18:27:23	2023-12-26 18:42:55	NULL	utf-8	NULL	NULL ============================================
q1	17717	4105	4078	4078
q2	q3	10856	1384	829	829
q4	4681	476	353	353
q5	7637	2247	2134	2134
q6	239	170	137	137
q7	982	760	635	635
q8	9352	1724	1666	1666
q9	5140	4979	4954	4954
q10	6363	2221	1871	1871
q11	452	279	243	243
q12	635	430	288	288
q13	18116	3403	2732	2732
q14	276	255	237	237
q15	q16	828	779	707	707
q17	936	884	961	884
q18	7211	5897	5550	5550
q19	1312	1367	1053	1053
q20	539	399	257	257
q21	5849	2634	2304	2304
q22	438	359	303	303
Total cold run time: 99559 ms
Total hot run time: 31215 ms
----- Round 2, with runtime_filter_mode=off -----
orders	Doris	NULL	NULL	150000000	42	6422171781	NULL	22778155	NULL	NULL	2023-12-26 18:27:23	2023-12-26 18:42:55	NULL	utf-8	NULL	NULL ============================================
q1	4407	4391	4283	4283
q2	q3	4544	5016	4397	4397
q4	2125	2242	1410	1410
q5	4497	4351	4365	4351
q6	321	289	164	164
q7	2287	1915	1720	1720
q8	2554	2230	2227	2227
q9	8391	7966	8060	7966
q10	4868	4775	4361	4361
q11	597	417	406	406
q12	780	757	547	547
q13	3380	3649	3027	3027
q14	295	313	269	269
q15	q16	713	751	658	658
q17	1428	1314	1491	1314
q18	7848	7464	7349	7349
q19	1160	1154	1089	1089
q20	2238	2243	1974	1974
q21	5324	4697	4548	4548
q22	528	459	426	426
Total cold run time: 58285 ms
Total hot run time: 52486 ms

JNSimba · 2026-05-26T06:05:24Z

run buildall

hello-stephen · 2026-05-26T06:15:34Z

TPC-DS: Total hot run time: 172998 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 918be4b864583885cb43290315b60963efa4f209, data reload: false
query5	4320	672	520	520
query6	337	224	202	202
query7	4245	571	305	305
query8	325	245	226	226
query9	8852	4197	4174	4174
query10	460	339	303	303
query11	5847	2396	2205	2205
query12	184	133	128	128
query13	1262	616	451	451
query14	6154	5556	5260	5260
query14_1	4508	4496	4480	4480
query15	213	205	184	184
query16	1023	448	356	356
query17	1134	737	584	584
query18	2553	485	345	345
query19	215	221	162	162
query20	144	135	129	129
query21	223	135	126	126
query22	13624	13927	13480	13480
query23	17427	16501	16417	16417
query23_1	16432	16472	16377	16377
query24	7506	1773	1321	1321
query24_1	1323	1365	1304	1304
query25	559	475	421	421
query26	1323	338	177	177
query27	2656	563	344	344
query28	4459	2050	2010	2010
query29	1021	637	531	531
query30	312	235	203	203
query31	1159	1099	961	961
query32	89	80	86	80
query33	564	368	307	307
query34	1221	1164	644	644
query35	787	816	694	694
query36	1424	1426	1307	1307
query37	154	114	98	98
query38	3256	3158	3107	3107
query39	934	931	903	903
query39_1	893	887	884	884
query40	245	149	134	134
query41	72	70	68	68
query42	111	111	113	111
query43	339	337	311	311
query44 query45	218	209	203	203
query46	1087	1259	740	740
query47	2377	2325	2258	2258
query48	420	418	320	320
query49	659	517	405	405
query50	1047	365	265	265
query51	4415	4319	4182	4182
query52	108	115	97	97
query53	255	280	213	213
query54	335	302	274	274
query55	103	95	93	93
query56	312	326	323	323
query57	1429	1441	1343	1343
query58	302	289	276	276
query59	1568	1693	1461	1461
query60	340	346	316	316
query61	189	182	185	182
query62	705	719	574	574
query63	251	203	211	203
query64	2409	813	653	653
query65 query66	1717	479	374	374
query67	29786	29723	29438	29438
query68 query69	463	356	312	312
query70	1021	989	1018	989
query71	320	278	258	258
query72	3166	2728	2468	2468
query73	847	765	446	446
query74	5110	4980	4861	4861
query75	2684	2645	2273	2273
query76	2295	1150	790	790
query77	403	419	345	345
query78	12356	12542	11916	11916
query79	1453	1052	710	710
query80	651	554	465	465
query81	451	281	247	247
query82	1369	170	128	128
query83	355	276	251	251
query84	265	144	113	113
query85	881	543	461	461
query86	419	357	304	304
query87	3450	3397	3258	3258
query88	3692	2769	2749	2749
query89	458	388	345	345
query90	1887	190	185	185
query91	182	170	144	144
query92	84	79	77	77
query93	1517	1369	865	865
query94	541	368	318	318
query95	702	462	356	356
query96	1088	835	358	358
query97	2765	2741	2620	2620
query98	233	233	247	233
query99	1187	1163	1041	1041
Total cold run time: 255651 ms
Total hot run time: 172998 ms

hello-stephen · 2026-05-26T06:34:18Z

TPC-H: Total hot run time: 31835 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit a68a33eeca17b81c6b13ff89018f6f396b2d57c8, data reload: false
------ Round 1 ----------------------------------
orders	Doris	NULL	NULL	0	0	0	NULL	0	NULL	NULL	2023-12-26 18:27:23	2023-12-26 18:42:55	NULL	utf-8	NULL	NULL ============================================
q1	17648	4086	4026	4026
q2	q3	10823	1395	804	804
q4	4682	480	346	346
q5	7578	2290	2117	2117
q6	231	175	135	135
q7	911	809	650	650
q8	9426	1599	1610	1599
q9	5166	5016	4976	4976
q10	6370	2197	1880	1880
q11	445	281	243	243
q12	643	423	302	302
q13	18154	3478	2809	2809
q14	276	258	241	241
q15	q16	834	779	710	710
q17	999	966	1000	966
q18	7254	5802	5601	5601
q19	1179	1229	1151	1151
q20	613	451	311	311
q21	5907	2921	2651	2651
q22	599	384	317	317
Total cold run time: 99738 ms
Total hot run time: 31835 ms
----- Round 2, with runtime_filter_mode=off -----
orders	Doris	NULL	NULL	150000000	42	6422171781	NULL	22778155	NULL	NULL	2023-12-26 18:27:23	2023-12-26 18:42:55	NULL	utf-8	NULL	NULL ============================================
q1	4943	4950	4735	4735
q2	q3	4917	5294	4716	4716
q4	2284	2447	1509	1509
q5	5019	4793	4646	4646
q6	227	174	134	134
q7	1958	1767	1601	1601
q8	2506	2179	2191	2179
q9	8061	7483	7367	7367
q10	4791	4734	4265	4265
q11	548	425	407	407
q12	740	751	549	549
q13	2977	3345	2813	2813
q14	278	277	264	264
q15	q16	687	717	625	625
q17	1309	1278	1284	1278
q18	7579	6986	6942	6942
q19	1105	1097	1112	1097
q20	2245	2245	2006	2006
q21	5422	4738	4709	4709
q22	538	482	425	425
Total cold run time: 58134 ms
Total hot run time: 52267 ms

hello-stephen · 2026-05-26T06:45:19Z

TPC-DS: Total hot run time: 172440 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit a68a33eeca17b81c6b13ff89018f6f396b2d57c8, data reload: false
query5	4327	698	528	528
query6	341	217	207	207
query7	4237	568	316	316
query8	337	234	219	219
query9	8830	4114	4109	4109
query10	456	340	300	300
query11	5765	2615	2261	2261
query12	179	128	125	125
query13	1283	643	438	438
query14	6149	5539	5254	5254
query14_1	4560	4593	4566	4566
query15	216	213	185	185
query16	1055	487	418	418
query17	1174	749	623	623
query18	2504	497	371	371
query19	218	208	169	169
query20	139	135	134	134
query21	216	143	121	121
query22	13761	13606	13476	13476
query23	17410	16674	16356	16356
query23_1	16490	16559	16439	16439
query24	7542	1799	1320	1320
query24_1	1337	1321	1341	1321
query25	592	516	448	448
query26	1309	327	180	180
query27	2689	577	345	345
query28	4448	2043	2054	2043
query29	1002	652	530	530
query30	297	239	207	207
query31	1148	1092	965	965
query32	92	77	77	77
query33	564	361	329	329
query34	1219	1092	655	655
query35	768	789	708	708
query36	1379	1443	1252	1252
query37	158	109	91	91
query38	3236	3200	3060	3060
query39	921	926	908	908
query39_1	900	881	863	863
query40	239	145	124	124
query41	64	65	63	63
query42	113	105	111	105
query43	335	338	300	300
query44 query45	214	204	194	194
query46	1089	1197	721	721
query47	2369	2364	2211	2211
query48	421	393	315	315
query49	621	489	388	388
query50	958	364	271	271
query51	4468	4345	4217	4217
query52	105	107	98	98
query53	253	280	205	205
query54	311	275	262	262
query55	95	90	86	86
query56	312	302	302	302
query57	1459	1399	1343	1343
query58	295	270	270	270
query59	1609	1691	1466	1466
query60	326	338	322	322
query61	162	155	152	152
query62	702	652	596	596
query63	248	202	207	202
query64	2398	822	616	616
query65 query66	1716	487	371	371
query67	29906	29948	29035	29035
query68 query69	456	343	304	304
query70	1054	994	1007	994
query71	295	273	275	273
query72	3012	2741	2422	2422
query73	850	737	420	420
query74	5157	5010	4830	4830
query75	2687	2620	2287	2287
query76	2302	1159	779	779
query77	398	446	339	339
query78	12367	12549	11877	11877
query79	1475	1022	791	791
query80	644	531	445	445
query81	453	284	247	247
query82	1359	158	121	121
query83	364	293	258	258
query84	267	148	106	106
query85	908	544	448	448
query86	407	353	352	352
query87	3461	3387	3222	3222
query88	3626	2785	2744	2744
query89	448	391	347	347
query90	1921	187	180	180
query91	180	167	143	143
query92	84	88	73	73
query93	1646	1477	939	939
query94	557	357	304	304
query95	682	479	349	349
query96	1044	754	357	357
query97	2748	2755	2625	2625
query98	234	232	233	232
query99	1186	1156	1030	1030
Total cold run time: 255326 ms
Total hot run time: 172440 ms

…per-reader assignment

…n regression assertions

…lient

…m in cdc_client

…ession and drop stale comment

JNSimba · 2026-05-26T09:01:25Z

/review

JNSimba · 2026-05-26T09:01:30Z

run buildall

hello-stephen · 2026-05-26T09:43:31Z

TPC-H: Total hot run time: 31842 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit a31e7d7a0d4b6bab4467798ffb1011d73bdc4fb1, data reload: false
------ Round 1 ----------------------------------
orders	Doris	NULL	NULL	0	0	0	NULL	0	NULL	NULL	2023-12-26 18:27:23	2023-12-26 18:42:55	NULL	utf-8	NULL	NULL ============================================
q1	17645	4084	4092	4084
q2	q3	10787	1399	821	821
q4	4681	475	350	350
q5	7554	2250	2118	2118
q6	255	180	141	141
q7	917	774	637	637
q8	9395	1786	1640	1640
q9	6963	5027	5002	5002
q10	6425	2223	1879	1879
q11	434	273	244	244
q12	696	421	295	295
q13	18245	3460	2787	2787
q14	270	252	238	238
q15	q16	825	778	704	704
q17	1010	1008	930	930
q18	6822	5790	5644	5644
q19	1218	1420	1139	1139
q20	513	418	281	281
q21	5977	2756	2584	2584
q22	449	373	324	324
Total cold run time: 101081 ms
Total hot run time: 31842 ms
----- Round 2, with runtime_filter_mode=off -----
orders	Doris	NULL	NULL	150000000	42	6422171781	NULL	22778155	NULL	NULL	2023-12-26 18:27:23	2023-12-26 18:42:55	NULL	utf-8	NULL	NULL ============================================
q1	4945	4731	5001	4731
q2	q3	4967	5269	4631	4631
q4	2138	2220	1432	1432
q5	4792	4852	4670	4670
q6	233	176	135	135
q7	1829	1726	1607	1607
q8	2429	1982	1936	1936
q9	7421	7507	7560	7507
q10	4782	4713	4226	4226
q11	550	438	367	367
q12	741	744	542	542
q13	3071	3415	2868	2868
q14	271	291	252	252
q15	q16	675	710	604	604
q17	1302	1266	1265	1265
q18	7552	6960	6754	6754
q19	1130	1095	1078	1078
q20	2245	2234	1963	1963
q21	5293	4657	4517	4517
q22	529	460	414	414
Total cold run time: 56895 ms
Total hot run time: 51499 ms

hello-stephen · 2026-05-26T09:54:29Z

TPC-DS: Total hot run time: 170745 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit a31e7d7a0d4b6bab4467798ffb1011d73bdc4fb1, data reload: false
query5	4354	651	514	514
query6	336	225	202	202
query7	4226	536	320	320
query8	322	232	217	217
query9	8821	4080	4070	4070
query10	455	352	296	296
query11	5816	2574	2238	2238
query12	178	127	126	126
query13	1285	612	426	426
query14	6189	5517	5178	5178
query14_1	4469	4496	4471	4471
query15	209	209	188	188
query16	1030	449	422	422
query17	1122	718	597	597
query18	2521	466	353	353
query19	205	194	164	164
query20	133	130	130	130
query21	215	140	119	119
query22	13666	13602	13379	13379
query23	17374	16580	16181	16181
query23_1	16448	16410	16400	16400
query24	7498	1733	1318	1318
query24_1	1339	1327	1358	1327
query25	580	513	463	463
query26	1311	342	183	183
query27	2682	569	357	357
query28	4455	1982	1973	1973
query29	1046	664	517	517
query30	313	240	212	212
query31	1134	1077	958	958
query32	90	79	77	77
query33	561	385	311	311
query34	1172	1124	659	659
query35	763	816	706	706
query36	1445	1417	1268	1268
query37	156	106	88	88
query38	3242	3124	3081	3081
query39	928	922	912	912
query39_1	875	884	882	882
query40	253	148	134	134
query41	78	67	67	67
query42	114	111	109	109
query43	331	337	294	294
query44 query45	220	211	216	211
query46	1091	1200	756	756
query47	2437	2365	2228	2228
query48	400	414	304	304
query49	633	499	381	381
query50	994	350	247	247
query51	4369	4317	4238	4238
query52	109	107	93	93
query53	248	283	194	194
query54	316	275	247	247
query55	95	90	85	85
query56	309	291	309	291
query57	1449	1425	1362	1362
query58	303	277	271	271
query59	1588	1681	1442	1442
query60	321	325	297	297
query61	163	153	161	153
query62	695	654	591	591
query63	244	200	200	200
query64	2373	798	630	630
query65 query66	1707	491	361	361
query67	29716	29043	29527	29043
query68 query69	460	346	304	304
query70	1007	1024	961	961
query71	293	273	268	268
query72	3118	2962	2482	2482
query73	873	783	408	408
query74	5118	5008	4780	4780
query75	2684	2631	2272	2272
query76	2330	1170	775	775
query77	414	407	333	333
query78	12451	12450	11950	11950
query79	1494	1011	732	732
query80	652	543	463	463
query81	455	283	245	245
query82	1318	157	120	120
query83	359	273	242	242
query84	276	144	116	116
query85	893	552	456	456
query86	400	369	361	361
query87	3432	3381	3251	3251
query88	3585	2731	2721	2721
query89	436	403	346	346
query90	1914	181	183	181
query91	173	167	144	144
query92	78	82	71	71
query93	1519	1467	900	900
query94	552	365	277	277
query95	684	402	352	352
query96	1085	805	331	331
query97	2747	2747	2625	2625
query98	249	229	232	229
query99	1153	1152	1033	1033
Total cold run time: 254618 ms
Total hot run time: 170745 ms

JNSimba · 2026-05-26T10:08:21Z

run valut_p0

github-actions

I found one blocking correctness issue in the new server_id validation.

Critical checkpoint conclusions:

Goal/test: The PR aims to support configurable MySQL CDC server_id and validate invalid ranges earlier. It covers initial snapshot paths, but the validation now rejects valid binlog-only configurations, and there is no test for offset=latest/earliest/specific offset with snapshot_parallelism > 1.
Scope/clarity: The change is focused, but the shared validation/runtime resolver uses snapshot_parallelism without considering startup mode.
Concurrency/lifecycle: The snapshot reader path uses per-subtask server IDs for parallel readers; no additional lock/lifecycle issue found in the reviewed change.
Config/compatibility: server_id is a new source property; existing jobs without it still derive IDs. The reported issue is a behavior regression for valid non-snapshot jobs that set snapshot_parallelism and a single explicit server_id.
Parallel paths: CREATE/TVF/runtime paths all share or mirror the same over-strict width requirement, so the problem affects analysis and CDC client execution.
Tests: Added tests cover malformed values and initial-mode happy paths, but not binlog-only modes where only one server_id is needed.
Observability/performance/transaction/persistence: No additional issue found for these checkpoints.

User focus: No additional user-provided review focus was specified.

hello-stephen · 2026-05-26T10:30:18Z

FE UT Coverage Report

Increment line coverage 71.11% (32/45) 🎉
Increment coverage report
Complete coverage report

JNSimba · 2026-05-26T10:51:22Z

run vault_p0

hello-stephen · 2026-05-26T11:56:59Z

FE Regression Coverage Report

Increment line coverage 3.10% (4/129) 🎉
Increment coverage report
Complete coverage report

JNSimba · 2026-05-27T04:50:45Z

run vault_p0

liaoxin01

LGTM

github-actions · 2026-05-28T06:52:56Z

PR approved by at least one committer and no changes requested.

already fix

#63404 #63471 #63480 #63490 #63514 #63618 (#63812) Cherry-picked from: - #63079 [improve](streaming-job) async chunk splitting for cdc source job - #63404 [test](streaming-job) refine cdc data-type and boundary regression cases for mysql/pg - #63471 [regression-test](streaming-job) add cdc cases (composite/concurrent-dml/id-gap/decimal/datetime pk) and fix split-bound java.time deserialize - #63480 [fix](streaming-job) misc fixes for typo/log/validation/visibility - #63402 [fix](streaming-job) drop neighbour-table rows leaked by JDBC LIKE wildcards in JdbcPostgreSQLClient - #63514 [regression-test](streaming-job) add cdc operational cases for offset modes and pg slot lifecycle - #63618 [fix](streaming-job) fix postgres historical-date timestamp handling in cdc-client - #63490 [improve](streaming-job) support user-specified mysql server_id with per-reader assignment

…per-reader assignment (apache#63490) ## Summary - Add an optional `server_id` source property for MySQL CDC streaming jobs. Accepts a single value (e.g. `5400`) or a range (e.g. `5400-5408`). When unset, the value is derived from the jobId hash so existing jobs keep their current server_id when `snapshot_parallelism = 1`. - Fix a latent collision: when `snapshot_parallelism > 1` and source-side DML happens during snapshot, all parallel `SnapshotSplitReader` instances previously shared the same server_id and their backfill BinaryLogClient connections kicked each other out of MySQL's dump-thread slot, dropping binlog events between low and high watermark. Each subtask now gets a distinct server_id from the resolved range; the single binlog reader uses the range start. - Cross-field check: reject `server_id` range width smaller than `snapshot_parallelism` at job startup with a clear fix-it suggestion.

JNSimba force-pushed the feat/streaming-job-mysql-server-id branch from 1315fab to 5696cc2Compare May 21, 2026 11:01

JNSimba requested a review from Copilot May 21, 2026 11:01

Copilotstarted reviewing on behalf of JNSimba May 21, 2026 11:01 View session

CopilotAI reviewed May 21, 2026
View reviewed changes

Comment threadfs_brokers/cdc_client/src/main/java/org/apache/doris/cdcclient/utils/ConfigUtil.java Outdated
Comment threadregression-test/suites/job_p0/streaming_job/cdc/test_streaming_mysql_job_server_id.groovy Outdated

github-actionsBot requested changes May 21, 2026
View reviewed changes

Comment thread...rc/main/java/org/apache/doris/job/extensions/insert/streaming/DataSourceConfigValidator.java

github-actionsBot requested changes May 22, 2026
View reviewed changes

Comment thread...rc/main/java/org/apache/doris/job/extensions/insert/streaming/DataSourceConfigValidator.java

github-actionsBot requested changes May 25, 2026
View reviewed changes

JNSimba added 4 commits May 26, 2026 16:50

[improve](streaming-job) support user-specified mysql server_id with …
ba1eb1c
…per-reader assignment

[improve](streaming-job) validate server_id at reader init and tighte…
4138a93
…n regression assertions

[improve](streaming-job) move server_id validation from cdc_client to FE
49006e8

[improve](streaming-job) drop redundant server_id validation in cdc_c…
85b2af7
…lient

JNSimba added 3 commits May 26, 2026 16:57

[improve](streaming-job) reject server_id width < snapshot_parallelis…
cc6eeae
…m in cdc_client

fix style
ef58720

[improve](streaming-job) add binlog increment check to server_id regr…
a31e7d7
…ession and drop stale comment

JNSimba force-pushed the feat/streaming-job-mysql-server-id branch from a68a33e to a31e7d7Compare May 26, 2026 08:58

github-actionsBot previously requested changes May 26, 2026
View reviewed changes

Comment thread...rc/main/java/org/apache/doris/job/extensions/insert/streaming/DataSourceConfigValidator.java

liaoxin01 approved these changes May 28, 2026
View reviewed changes

github-actionsBot added the approved Indicates a PR has been approved by one committer. label May 28, 2026

JNSimba merged commit 7c4dfe9 into apache:masterMay 28, 2026
32 checks passed

JNSimba mentioned this pull request May 28, 2026
branch-4.1: [pick](streamingjob) pick streaming-job batch #63079 #63402 #63404 #63471 #63480 #63490 #63514 #63618 #63812
Merged

yiguolei added the dev/4.1.2-merged label May 29, 2026

yiguolei mentioned this pull request Jun 14, 2026
4.1.2 Release Notes #64485
Closed

Uh oh!

Conversation

JNSimba commented May 21, 2026

Summary

Test plan

Uh oh!

hello-stephen commented May 21, 2026

Uh oh!

JNSimba commented May 21, 2026

Uh oh!

JNSimba commented May 21, 2026

Uh oh!

CopilotAI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

github-actionsBot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

hello-stephen commented May 21, 2026

Uh oh!

hello-stephen commented May 21, 2026

Uh oh!

hello-stephen commented May 21, 2026

FE UT Coverage Report

Uh oh!

hello-stephen commented May 21, 2026

FE Regression Coverage Report

Uh oh!

JNSimba commented May 22, 2026

Uh oh!

JNSimba commented May 22, 2026

Uh oh!

github-actionsBot left a comment

Choose a reason for hiding this comment

Uh oh!

JNSimba commented May 22, 2026

Uh oh!

github-actionsBot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

JNSimba commented May 25, 2026

Uh oh!

github-actionsBot left a comment

Choose a reason for hiding this comment

Uh oh!

JNSimba commented May 26, 2026

Uh oh!

hello-stephen commented May 26, 2026

FE UT Coverage Report

Uh oh!

hello-stephen commented May 26, 2026

Uh oh!

JNSimba commented May 26, 2026

Uh oh!

hello-stephen commented May 26, 2026

Uh oh!

hello-stephen commented May 26, 2026

Uh oh!

hello-stephen commented May 26, 2026

Uh oh!

JNSimba commented May 26, 2026

Uh oh!

JNSimba commented May 26, 2026

Uh oh!

hello-stephen commented May 26, 2026

Uh oh!

hello-stephen commented May 26, 2026

Uh oh!

JNSimba commented May 26, 2026

Uh oh!

github-actionsBot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!