[Fix](StreamingJob) fix create table issues when create streaming job by JNSimba · Pull Request #59828 · apache/doris

JNSimba · 2026-01-13T09:02:28Z

What problem does this PR solve?

Related PR: #58898

The length of varchar needs to be multiplied by 3 when creating the table.
Columns are ordered according to the primary key.
Unsupported column types will result in an error.

Release note

None

Check List (For Author)

Test
- Regression test
- Unit Test
- Manual test (add detailed scripts or steps below)
- No need to test or manual test. Explain why:
  - This is a refactor/code format and no logic has been changed.
  - Previous test can cover this change.
  - No code files have been changed.
  - Other reason
Behavior changed:
- No.
- Yes.
Does this need documentation?
- No.
- Yes.

Check List (For Reviewer who merge this PR)

Confirm the release note
Confirm test cases
Confirm document
Add branch pick label

hello-stephen · 2026-01-13T09:02:43Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

What problem was fixed (it's best to include specific error reporting information). How it was fixed.
Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
What features were added. Why was this function added?
Which code was refactored and why was this part of the code refactored?
Which functions were optimized and what is the difference before and after the optimization?

JNSimba · 2026-01-13T10:36:47Z

run buildall

doris-robot · 2026-01-13T11:06:45Z

TPC-H: Total hot run time: 31878 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 6b1e40687a3eade2cd3ffcf42dc4287c64a40430, data reload: false
------ Round 1 ----------------------------------
q1	17610	4112	4023	4023
q2	2031	362	238	238
q3	10171	1270	731	731
q4	10236	921	325	325
q5	7535	2042	1840	1840
q6	195	167	138	138
q7	930	781	653	653
q8	9263	1392	1188	1188
q9	4833	4622	4470	4470
q10	6744	1798	1428	1428
q11	520	320	266	266
q12	696	737	587	587
q13	17747	3839	3061	3061
q14	290	307	287	287
q15	571	511	501	501
q16	682	674	633	633
q17	635	777	519	519
q18	6700	6459	6844	6459
q19	1130	1063	650	650
q20	426	389	266	266
q21	3168	2679	2541	2541
q22	1157	1074	1078	1074
Total cold run time: 103270 ms
Total hot run time: 31878 ms
----- Round 2, with runtime_filter_mode=off -----
q1	4320	4353	4282	4282
q2	376	463	321	321
q3	2232	2818	2268	2268
q4	1460	1878	1443	1443
q5	4512	4262	4273	4262
q6	217	169	123	123
q7	1962	1939	1790	1790
q8	2530	2498	2489	2489
q9	7361	7191	7140	7140
q10	2519	2788	2301	2301
q11	557	487	445	445
q12	720	752	663	663
q13	3585	3949	3411	3411
q14	306	308	292	292
q15	542	503	510	503
q16	655	678	624	624
q17	1083	1221	1301	1221
q18	7519	7461	7298	7298
q19	818	756	809	756
q20	1873	1972	1804	1804
q21	4423	4237	4134	4134
q22	1056	1035	978	978
Total cold run time: 50626 ms
Total hot run time: 48548 ms

doris-robot · 2026-01-13T11:17:38Z

TPC-DS: Total hot run time: 175143 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 6b1e40687a3eade2cd3ffcf42dc4287c64a40430, data reload: false
query5	4818	647	510	510
query6	359	260	237	237
query7	4226	463	268	268
query8	388	284	287	284
query9	8798	2933	2902	2902
query10	545	398	350	350
query11	15256	15284	14850	14850
query12	184	124	122	122
query13	1284	509	406	406
query14	6535	3053	2816	2816
query14_1	2716	2737	2696	2696
query15	209	200	182	182
query16	1036	504	496	496
query17	1138	753	668	668
query18	2708	449	356	356
query19	241	230	208	208
query20	130	121	126	121
query21	217	139	121	121
query22	3869	4056	3843	3843
query23	15922	15532	15328	15328
query23_1	15380	15450	15374	15374
query24	7153	1552	1159	1159
query24_1	1180	1143	1209	1143
query25	584	496	449	449
query26	1250	270	161	161
query27	2749	452	276	276
query28	4536	2182	2184	2182
query29	814	616	474	474
query30	310	252	215	215
query31	836	636	578	578
query32	88	79	76	76
query33	555	383	332	332
query34	928	877	536	536
query35	747	772	704	704
query36	881	938	856	856
query37	129	103	94	94
query38	2731	2731	2699	2699
query39	780	752	746	746
query39_1	734	725	709	709
query40	228	148	130	130
query41	81	79	75	75
query42	110	110	108	108
query43	448	462	418	418
query44	1334	749	755	749
query45	190	194	184	184
query46	830	943	572	572
query47	1493	1494	1429	1429
query48	314	331	253	253
query49	638	456	370	370
query50	607	288	223	223
query51	3817	3850	3755	3755
query52	109	111	104	104
query53	312	334	282	282
query54	323	295	288	288
query55	80	81	80	80
query56	331	342	344	342
query57	1026	1051	927	927
query58	298	283	284	283
query59	1929	2247	2021	2021
query60	370	360	347	347
query61	198	198	198	198
query62	399	385	308	308
query63	312	276	275	275
query64	5132	1622	1336	1336
query65	3820	3712	3797	3712
query66	1454	475	391	391
query67	15096	15320	14599	14599
query68	4648	1003	723	723
query69	531	388	360	360
query70	1081	962	910	910
query71	381	329	303	303
query72	6312	3634	3755	3634
query73	762	721	308	308
query74	8815	8776	8585	8585
query75	2837	2833	2468	2468
query76	3452	1066	656	656
query77	530	384	342	342
query78	9884	10013	9139	9139
query79	996	877	571	571
query80	1245	624	559	559
query81	571	276	237	237
query82	411	156	117	117
query83	375	271	260	260
query84	253	121	110	110
query85	1029	645	589	589
query86	406	305	306	305
query87	2837	2876	2737	2737
query88	2951	2164	2191	2164
query89	407	352	339	339
query90	1959	178	169	169
query91	201	197	163	163
query92	83	82	76	76
query93	1039	881	533	533
query94	670	350	331	331
query95	614	353	399	353
query96	567	469	201	201
query97	2328	2386	2305	2305
query98	224	203	206	203
query99	613	614	544	544
Total cold run time: 252551 ms
Total hot run time: 175143 ms

Copilot

Pull request overview

This PR fixes three issues when creating tables for streaming jobs:

VARCHAR column lengths are now multiplied by 3 to accommodate multi-byte character encodings
Columns are reordered to place primary key columns first, in the order specified
Unsupported column types now trigger an error with a clear message

Changes:

Enhanced getColumns method to multiply varchar lengths by 3, validate against unsupported types, and sort columns by primary key order
Added comprehensive unit tests covering varchar conversion, primary key sorting, and edge cases
Updated regression test to verify varchar length multiplication (200 * 3 = 600)
Added logging and toString support for commit offset operations

Reviewed changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 6 comments.

File	Description
fe/fe-core/src/main/java/org/apache/doris/job/util/StreamingJobUtils.java	Core logic changes: varchar length multiplication by 3, unsupported type validation, and primary key-based column sorting
fe/fe-core/src/test/java/org/apache/doris/job/util/StreamingJobUtilsTest.java	New comprehensive unit tests for getColumns method covering varchar conversion, primary key sorting, and edge cases
regression-test/suites/job_p0/streaming_job/cdc/test_streaming_mysql_job.groovy	Updated assertion to verify varchar(200) becomes varchar(600), added table3 cleanup
fe/fe-core/src/main/java/org/apache/doris/httpv2/rest/StreamingJobAction.java	Added logging for offset commits and @tostring annotation for better debugging

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

JNSimba · 2026-01-13T11:44:13Z

run buildall

JNSimba · 2026-01-13T11:49:42Z

run buildall

doris-robot · 2026-01-13T12:20:53Z

TPC-H: Total hot run time: 31030 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 6b72a465c1899a37ee3762a26302d9319df3e967, data reload: false
------ Round 1 ----------------------------------
q1	17615	4265	4038	4038
q2	2076	353	256	256
q3	10116	1314	720	720
q4	10217	857	310	310
q5	7503	2078	1803	1803
q6	183	174	141	141
q7	922	775	639	639
q8	9262	1366	1139	1139
q9	4847	4666	4520	4520
q10	6798	1777	1415	1415
q11	515	302	289	289
q12	684	759	580	580
q13	17794	3799	3086	3086
q14	291	309	274	274
q15	594	513	513	513
q16	689	677	635	635
q17	650	789	503	503
q18	6612	6311	6217	6217
q19	1229	968	594	594
q20	378	356	236	236
q21	2993	2525	2166	2166
q22	1020	1003	956	956
Total cold run time: 102988 ms
Total hot run time: 31030 ms
----- Round 2, with runtime_filter_mode=off -----
q1	4109	4064	4061	4061
q2	327	395	303	303
q3	2116	2597	2240	2240
q4	1312	1749	1330	1330
q5	4081	4010	4060	4010
q6	200	165	131	131
q7	1867	1826	1690	1690
q8	2771	2584	2400	2400
q9	7197	7156	7013	7013
q10	2488	2630	2331	2331
q11	572	480	456	456
q12	754	802	637	637
q13	3676	4145	3275	3275
q14	296	311	280	280
q15	571	612	530	530
q16	635	666	668	666
q17	1184	1311	1391	1311
q18	8152	7795	7636	7636
q19	913	851	879	851
q20	1991	2153	1924	1924
q21	4781	4322	4045	4045
q22	1055	1030	965	965
Total cold run time: 51048 ms
Total hot run time: 48085 ms

doris-robot · 2026-01-13T12:31:40Z

TPC-DS: Total hot run time: 173039 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 6b72a465c1899a37ee3762a26302d9319df3e967, data reload: false
query5	4438	621	491	491
query6	341	233	221	221
query7	4219	456	257	257
query8	342	259	248	248
query9	8705	2880	2856	2856
query10	492	400	325	325
query11	15151	15138	14725	14725
query12	183	120	126	120
query13	1291	482	384	384
query14	6139	2986	2736	2736
query14_1	2665	2648	2641	2641
query15	210	205	175	175
query16	1004	474	468	468
query17	1103	689	589	589
query18	2527	439	334	334
query19	224	226	197	197
query20	126	123	120	120
query21	213	139	117	117
query22	3905	4053	3986	3986
query23	15929	15716	15290	15290
query23_1	15567	15610	15424	15424
query24	7216	1557	1174	1174
query24_1	1177	1168	1175	1168
query25	573	482	430	430
query26	1251	271	156	156
query27	2763	446	268	268
query28	4588	2192	2177	2177
query29	801	545	488	488
query30	309	246	215	215
query31	844	635	553	553
query32	83	80	72	72
query33	531	368	312	312
query34	895	870	526	526
query35	718	764	701	701
query36	945	888	853	853
query37	134	106	84	84
query38	2684	2717	2651	2651
query39	765	740	726	726
query39_1	722	716	703	703
query40	214	130	117	117
query41	70	61	62	61
query42	103	106	107	106
query43	431	472	419	419
query44	1344	742	730	730
query45	182	178	179	178
query46	835	945	577	577
query47	1366	1471	1315	1315
query48	297	311	225	225
query49	605	424	331	331
query50	601	274	203	203
query51	3743	3768	3899	3768
query52	105	107	97	97
query53	288	321	273	273
query54	287	267	255	255
query55	79	78	73	73
query56	318	313	320	313
query57	1029	1011	955	955
query58	275	269	259	259
query59	2162	2207	2072	2072
query60	371	328	311	311
query61	156	154	149	149
query62	412	348	308	308
query63	304	269	274	269
query64	4902	1276	960	960
query65	3779	3779	3659	3659
query66	1428	430	320	320
query67	15330	15957	15138	15138
query68	5780	1006	703	703
query69	497	344	323	323
query70	1014	967	946	946
query71	349	307	284	284
query72	5733	3374	3382	3374
query73	746	711	311	311
query74	8746	8789	8612	8612
query75	2817	2809	2448	2448
query76	3461	1058	636	636
query77	504	388	312	312
query78	9703	9881	9082	9082
query79	1210	875	556	556
query80	688	580	489	489
query81	514	259	228	228
query82	404	143	109	109
query83	268	250	243	243
query84	254	118	96	96
query85	865	511	443	443
query86	335	324	295	295
query87	2872	2833	2738	2738
query88	2891	2101	2098	2098
query89	379	345	333	333
query90	1969	180	158	158
query91	170	160	138	138
query92	76	71	68	68
query93	1076	910	532	532
query94	584	317	310	310
query95	574	383	316	316
query96	550	459	195	195
query97	2326	2367	2299	2299
query98	216	206	196	196
query99	587	593	506	506
Total cold run time: 250036 ms
Total hot run time: 173039 ms

JNSimba · 2026-01-13T13:14:50Z

run buildall

doris-robot · 2026-01-13T14:17:02Z

TPC-H: Total hot run time: 31423 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit b1ac6906752030419796cb2e5b651a8312268a2b, data reload: false
------ Round 1 ----------------------------------
q1	17609	4259	4072	4072
q2	2083	353	237	237
q3	10170	1288	709	709
q4	10225	916	313	313
q5	7544	2108	1807	1807
q6	191	174	142	142
q7	922	790	655	655
q8	9265	1392	1149	1149
q9	4997	4708	4620	4620
q10	6783	1819	1425	1425
q11	488	303	297	297
q12	676	743	588	588
q13	17783	3814	3161	3161
q14	286	290	281	281
q15	594	514	507	507
q16	692	695	627	627
q17	656	876	415	415
q18	6667	6495	6317	6317
q19	1107	968	618	618
q20	403	355	252	252
q21	3026	2472	2266	2266
q22	1025	1015	965	965
Total cold run time: 103192 ms
Total hot run time: 31423 ms
----- Round 2, with runtime_filter_mode=off -----
q1	4105	4052	4064	4052
q2	343	413	335	335
q3	2082	2554	2224	2224
q4	1318	1739	1319	1319
q5	4066	3997	4032	3997
q6	209	169	129	129
q7	1924	1791	1674	1674
q8	2861	2439	2434	2434
q9	7223	7367	7206	7206
q10	2489	2745	2243	2243
q11	606	483	456	456
q12	753	794	640	640
q13	3586	4109	3355	3355
q14	292	322	372	322
q15	560	495	499	495
q16	751	674	635	635
q17	1175	1288	1432	1288
q18	8128	7833	7791	7791
q19	910	882	870	870
q20	2101	2049	1890	1890
q21	4863	4423	4341	4341
q22	1092	1003	943	943
Total cold run time: 51437 ms
Total hot run time: 48639 ms

doris-robot · 2026-01-13T14:27:49Z

TPC-DS: Total hot run time: 172158 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit b1ac6906752030419796cb2e5b651a8312268a2b, data reload: false
query5	4376	597	482	482
query6	332	220	214	214
query7	4215	467	249	249
query8	333	246	234	234
query9	8723	2879	2874	2874
query10	531	391	319	319
query11	15245	15083	14822	14822
query12	171	113	113	113
query13	1256	480	358	358
query14	6094	2960	2718	2718
query14_1	2609	2627	2594	2594
query15	191	188	174	174
query16	1018	494	461	461
query17	1090	689	602	602
query18	2448	451	337	337
query19	232	235	198	198
query20	123	116	116	116
query21	212	138	118	118
query22	3923	4072	4021	4021
query23	15927	15489	15465	15465
query23_1	15391	15621	15485	15485
query24	7214	1562	1128	1128
query24_1	1170	1169	1186	1169
query25	578	481	444	444
query26	1248	266	154	154
query27	2771	445	267	267
query28	4604	2139	2136	2136
query29	793	549	461	461
query30	307	250	214	214
query31	798	650	557	557
query32	82	80	77	77
query33	542	366	310	310
query34	924	880	520	520
query35	713	778	686	686
query36	876	971	859	859
query37	131	100	86	86
query38	2757	2718	2655	2655
query39	785	763	740	740
query39_1	752	703	728	703
query40	232	155	124	124
query41	71	68	69	68
query42	109	109	104	104
query43	431	466	408	408
query44	1367	761	745	745
query45	189	191	180	180
query46	817	933	576	576
query47	1447	1516	1299	1299
query48	302	331	240	240
query49	630	440	358	358
query50	615	274	213	213
query51	3768	3866	3721	3721
query52	112	110	102	102
query53	300	324	278	278
query54	318	303	305	303
query55	83	82	79	79
query56	322	325	323	323
query57	1018	1020	1005	1005
query58	268	260	261	260
query59	2027	2201	2142	2142
query60	326	334	311	311
query61	152	154	151	151
query62	380	355	299	299
query63	299	262	275	262
query64	4948	1333	993	993
query65	3854	3782	3713	3713
query66	1448	438	321	321
query67	15175	14669	15152	14669
query68	6525	971	689	689
query69	514	353	322	322
query70	1066	967	921	921
query71	378	304	289	289
query72	5748	3221	2588	2588
query73	722	631	294	294
query74	8815	8967	8557	8557
query75	2806	2790	2465	2465
query76	3385	1061	661	661
query77	529	379	296	296
query78	9711	9755	9234	9234
query79	1558	870	549	549
query80	673	582	473	473
query81	525	267	230	230
query82	213	154	108	108
query83	256	257	241	241
query84	255	115	95	95
query85	897	519	458	458
query86	389	321	290	290
query87	2869	2845	2740	2740
query88	2883	2092	2070	2070
query89	376	349	329	329
query90	2216	175	160	160
query91	167	157	137	137
query92	89	74	70	70
query93	1244	907	526	526
query94	574	333	302	302
query95	586	390	322	322
query96	553	469	192	192
query97	2315	2362	2294	2294
query98	239	205	198	198
query99	602	582	491	491
Total cold run time: 251297 ms
Total hot run time: 172158 ms

JNSimba · 2026-01-13T15:23:10Z

run cloud_p0

sollhui

LGTM

github-actions · 2026-01-14T02:03:04Z

PR approved by anyone and no changes requested.

github-actions · 2026-01-14T02:16:21Z

PR approved by at least one committer and no changes requested.

…#59828) ### What problem does this PR solve? Related PR: #58898 1. The length of varchar needs to be multiplied by 3 when creating the table. 2. Columns are ordered according to the primary key. 3. Unsupported column types will result in an error.

…treaming job #59828 (#59853) Cherry-picked from #59828 Co-authored-by: wudi <wudi@selectdb.com>

…treaming job apache#59828 (apache#59853) Cherry-picked from apache#59828 Co-authored-by: wudi <wudi@selectdb.com>

fix create table
40556e9

fix
6b1e406

JNSimba added the dev/4.0.x label Jan 13, 2026

JNSimba requested a review from Copilot January 13, 2026 11:22

Copilotstarted reviewing on behalf of JNSimba January 13, 2026 11:23 View session

CopilotAI reviewed Jan 13, 2026
View reviewed changes

fix
b5d600c

fix codestyle
6b72a46

fix case
b1ac690

sollhui approved these changes Jan 14, 2026
View reviewed changes

github-actionsBot added the reviewed label Jan 14, 2026

CalvinKirs approved these changes Jan 14, 2026
View reviewed changes

github-actionsBot added the approved Indicates a PR has been approved by one committer. label Jan 14, 2026

JNSimba merged commit 410e531 into apache:masterJan 14, 2026
27 of 29 checks passed

github-actionsBot mentioned this pull request Jan 14, 2026
branch-4.0: [Fix](StreamingJob) fix create table issues when create streaming job #59828 #59853
Merged

yiguolei pushed a commit that referenced this pull request Jan 14, 2026
branch-4.0: [Fix](StreamingJob) fix create table issues when create s…
d556b6e
…treaming job #59828 (#59853) Cherry-picked from #59828 Co-authored-by: wudi <wudi@selectdb.com>

yiguolei added dev/4.0.3-merged and removed dev/4.0.x labels Jan 14, 2026

ybtsdst pushed a commit to ybtsdst/doris that referenced this pull request Feb 27, 2026
branch-4.0: [Fix](StreamingJob) fix create table issues when create s…
f0c73ff
…treaming job apache#59828 (apache#59853) Cherry-picked from apache#59828 Co-authored-by: wudi <wudi@selectdb.com>

Uh oh!

Conversation

JNSimba commented Jan 13, 2026• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What problem does this PR solve?

Release note

Check List (For Author)

Check List (For Reviewer who merge this PR)

Uh oh!

hello-stephen commented Jan 13, 2026

Uh oh!

JNSimba commented Jan 13, 2026

Uh oh!

doris-robot commented Jan 13, 2026

Uh oh!

doris-robot commented Jan 13, 2026

Uh oh!

CopilotAI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

JNSimba commented Jan 13, 2026

Uh oh!

JNSimba commented Jan 13, 2026

Uh oh!

doris-robot commented Jan 13, 2026

Uh oh!

doris-robot commented Jan 13, 2026

Uh oh!

JNSimba commented Jan 13, 2026

Uh oh!

doris-robot commented Jan 13, 2026

Uh oh!

doris-robot commented Jan 13, 2026

Uh oh!

JNSimba commented Jan 13, 2026

Uh oh!

sollhui left a comment

Choose a reason for hiding this comment

Uh oh!

github-actionsBot commented Jan 14, 2026

Uh oh!

github-actionsBot commented Jan 14, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

JNSimba commented Jan 13, 2026•
edited
Loading