[improvement](jdbc catalog) support jdbc external catalog insert stmt in nereids by zy-kkk · Pull Request #39813 · apache/doris

zy-kkk · 2024-08-22T16:12:39Z

No description provided.

doris-robot · 2024-08-22T16:12:44Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

zy-kkk · 2024-08-27T07:20:48Z

run buildall

doris-robot · 2024-08-27T08:04:40Z

TPC-H: Total hot run time: 37773 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 559b20ca98e36b410be586a2aa6b0ea131b076a7, data reload: false
------ Round 1 ----------------------------------
q1	17602	4412	4275	4275
q2	2021	180	177	177
q3	11890	953	1046	953
q4	10522	801	700	700
q5	7768	2829	2756	2756
q6	224	136	136	136
q7	948	616	597	597
q8	9313	2070	2087	2070
q9	7215	6560	6568	6560
q10	6985	2230	2292	2230
q11	451	236	242	236
q12	393	222	222	222
q13	18042	3018	3022	3018
q14	279	235	240	235
q15	520	481	502	481
q16	509	398	391	391
q17	981	676	716	676
q18	7320	6912	6847	6847
q19	1399	1132	1095	1095
q20	667	336	334	334
q21	3807	2791	2960	2791
q22	1136	1028	993	993
Total cold run time: 109992 ms
Total hot run time: 37773 ms
----- Round 2, with runtime_filter_mode=off -----
q1	4326	4308	4305	4305
q2	380	265	267	265
q3	2900	2637	2694	2637
q4	1910	1651	1633	1633
q5	5541	5673	5690	5673
q6	241	144	138	138
q7	2199	1840	1813	1813
q8	3306	3400	3382	3382
q9	8888	8829	8876	8829
q10	3569	3401	3372	3372
q11	616	522	527	522
q12	822	673	672	672
q13	12771	3242	3224	3224
q14	320	291	296	291
q15	523	512	485	485
q16	477	447	435	435
q17	1828	1545	1511	1511
q18	8182	7777	8199	7777
q19	1766	1790	1424	1424
q20	2181	1915	1943	1915
q21	5849	5456	5470	5456
q22	1186	1061	1024	1024
Total cold run time: 69781 ms
Total hot run time: 56783 ms

doris-robot · 2024-08-27T08:16:53Z

TPC-DS: Total hot run time: 190838 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 559b20ca98e36b410be586a2aa6b0ea131b076a7, data reload: false
query1	1245	886	868	868
query2	6173	1931	1846	1846
query3	10732	3910	4029	3910
query4	60284	26900	23212	23212
query5	5347	500	501	500
query6	410	156	167	156
query7	5764	295	285	285
query8	295	202	207	202
query9	8825	2547	2494	2494
query10	492	269	251	251
query11	18198	14807	15402	14807
query12	147	101	100	100
query13	1547	394	410	394
query14	11332	6818	7236	6818
query15	287	177	179	177
query16	7643	495	487	487
query17	1152	571	591	571
query18	2041	294	285	285
query19	303	151	144	144
query20	115	107	105	105
query21	222	105	102	102
query22	4553	4478	4613	4478
query23	34224	33390	33320	33320
query24	5935	2842	2866	2842
query25	552	401	397	397
query26	680	159	154	154
query27	1757	274	279	274
query28	3820	2069	2061	2061
query29	722	427	424	424
query30	231	156	149	149
query31	938	765	755	755
query32	86	57	59	57
query33	477	299	292	292
query34	861	471	471	471
query35	857	732	724	724
query36	1053	935	932	932
query37	141	84	80	80
query38	4007	3869	3859	3859
query39	1483	1422	1433	1422
query40	201	119	121	119
query41	49	48	46	46
query42	112	98	99	98
query43	517	492	478	478
query44	1081	742	758	742
query45	204	169	167	167
query46	1086	732	738	732
query47	1853	1784	1811	1784
query48	380	306	307	306
query49	776	437	456	437
query50	819	426	408	408
query51	7216	7112	7035	7035
query52	98	87	88	87
query53	252	182	182	182
query54	576	459	456	456
query55	78	75	80	75
query56	368	255	246	246
query57	1191	1063	1089	1063
query58	226	218	233	218
query59	2971	2751	2876	2751
query60	291	267	272	267
query61	102	102	127	102
query62	785	672	657	657
query63	217	182	175	175
query64	3223	1761	1745	1745
query65	3162	3140	3121	3121
query66	702	324	331	324
query67	15454	15189	15103	15103
query68	3464	562	549	549
query69	396	274	283	274
query70	1106	1087	1028	1028
query71	364	280	281	280
query72	6156	2322	2065	2065
query73	754	321	337	321
query74	9141	8781	8868	8781
query75	3363	2679	2629	2629
query76	1446	996	978	978
query77	561	331	380	331
query78	10013	9408	8982	8982
query79	1041	547	536	536
query80	714	520	499	499
query81	474	225	222	222
query82	233	130	137	130
query83	169	152	150	150
query84	258	75	75	75
query85	725	276	319	276
query86	307	310	273	273
query87	4285	4308	4291	4291
query88	3125	2310	2323	2310
query89	392	291	282	282
query90	1877	190	192	190
query91	121	100	98	98
query92	58	53	58	53
query93	1050	533	532	532
query94	629	302	289	289
query95	360	259	261	259
query96	592	265	265	265
query97	3200	3092	3091	3091
query98	220	198	205	198
query99	1525	1279	1272	1272
Total cold run time: 307040 ms
Total hot run time: 190838 ms

doris-robot · 2024-08-27T08:22:25Z

ClickBench: Total hot run time: 31.47 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 559b20ca98e36b410be586a2aa6b0ea131b076a7, data reload: false
query1	0.05	0.04	0.04
query2	0.08	0.05	0.04
query3	0.23	0.06	0.05
query4	1.68	0.09	0.08
query5	0.51	0.50	0.48
query6	1.14	0.76	0.72
query7	0.02	0.01	0.02
query8	0.06	0.04	0.04
query9	0.54	0.48	0.47
query10	0.54	0.53	0.53
query11	0.15	0.11	0.11
query12	0.16	0.12	0.11
query13	0.61	0.58	0.58
query14	0.77	0.81	0.77
query15	0.89	0.83	0.82
query16	0.37	0.37	0.38
query17	1.03	1.01	1.01
query18	0.21	0.21	0.20
query19	1.96	1.70	1.84
query20	0.01	0.01	0.00
query21	15.39	0.66	0.65
query22	4.04	6.81	2.67
query23	18.32	1.28	1.34
query24	2.17	0.23	0.22
query25	0.16	0.08	0.08
query26	0.26	0.18	0.18
query27	0.07	0.09	0.08
query28	13.28	1.01	1.00
query29	12.62	3.43	3.42
query30	0.24	0.05	0.06
query31	2.89	0.40	0.39
query32	3.24	0.48	0.47
query33	2.94	2.95	3.00
query34	17.08	4.39	4.43
query35	4.48	4.44	4.49
query36	0.67	0.49	0.49
query37	0.19	0.16	0.16
query38	0.16	0.14	0.16
query39	0.05	0.04	0.03
query40	0.16	0.14	0.12
query41	0.10	0.05	0.05
query42	0.06	0.04	0.04
query43	0.05	0.04	0.04
Total cold run time: 109.63 s
Total hot run time: 31.47 s

morrySnow · 2024-09-05T12:27:56Z

+ return new UnboundJdbcTableSink<>(nameParts, colNames, hints, partitions,
+ dmlCommandType, groupExpression, Optional.empty(), children.get(0));


with children should not inherit groupExpression use Optional.empty()

morrySnow · 2024-09-05T12:34:34Z

+ @Override
+ public Void visitPhysicalJdbcTableSink(
+ PhysicalJdbcTableSink<? extends Plan> jdbcTableSink, PlanContext context) {
+ if (connectContext != null && !connectContext.getSessionVariable().enableStrictConsistencyDml) {


i think jdbc sink should always use gather properties

morrySnow · 2024-09-05T12:36:54Z

+ dmlCommandType, Optional.empty(), Optional.empty(), plan);
 }
 // TODO: we need to support insert into other catalog
 try {


remove these force fallback code

zy-kkk · 2024-09-06T02:58:40Z

run buildall

doris-robot · 2024-09-06T03:18:45Z

TPC-H: Total hot run time: 38234 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 9da97246649e9f8ace5bca4477961508ceafce77, data reload: false
------ Round 1 ----------------------------------
q1	18062	4539	4415	4415
q2	2519	194	194	194
q3	10944	1126	1055	1055
q4	10882	749	700	700
q5	7774	2886	2837	2837
q6	235	140	138	138
q7	987	620	607	607
q8	9432	2047	2054	2047
q9	7078	6523	6543	6523
q10	7035	2191	2153	2153
q11	448	243	242	242
q12	400	230	223	223
q13	17988	3097	3093	3093
q14	290	239	252	239
q15	532	475	478	475
q16	512	439	429	429
q17	985	670	706	670
q18	7427	6853	6881	6853
q19	1401	1058	1043	1043
q20	677	355	338	338
q21	3982	2967	2990	2967
q22	1121	1013	993	993
Total cold run time: 110711 ms
Total hot run time: 38234 ms
----- Round 2, with runtime_filter_mode=off -----
q1	4397	4309	4306	4306
q2	391	275	281	275
q3	2869	2659	2680	2659
q4	1960	1648	1655	1648
q5	5424	5384	5422	5384
q6	218	130	131	130
q7	2104	1691	1735	1691
q8	3166	3332	3335	3332
q9	8400	8457	8373	8373
q10	3472	3226	3200	3200
q11	606	499	511	499
q12	798	590	659	590
q13	11214	3109	3113	3109
q14	323	274	276	274
q15	529	480	482	480
q16	528	464	469	464
q17	1808	1492	1490	1490
q18	7603	7432	7480	7432
q19	1708	1481	1521	1481
q20	2078	1818	1788	1788
q21	5468	5256	5177	5177
q22	1097	1048	1029	1029
Total cold run time: 66161 ms
Total hot run time: 54811 ms

doris-robot · 2024-09-06T03:30:24Z

TPC-DS: Total hot run time: 187543 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 9da97246649e9f8ace5bca4477961508ceafce77, data reload: false
query1	915	374	382	374
query2	6499	1986	1916	1916
query3	6657	211	214	211
query4	30337	23179	23275	23179
query5	4159	512	485	485
query6	275	172	168	168
query7	4586	299	297	297
query8	290	228	226	226
query9	8591	2518	2523	2518
query10	425	261	269	261
query11	17804	15050	14961	14961
query12	169	96	97	96
query13	1633	391	369	369
query14	9778	7057	6900	6900
query15	283	166	168	166
query16	7921	478	409	409
query17	1567	573	550	550
query18	1960	287	286	286
query19	209	139	141	139
query20	118	108	111	108
query21	212	103	104	103
query22	4448	4091	4098	4091
query23	34189	33346	33348	33346
query24	11241	3024	2821	2821
query25	649	383	380	380
query26	1178	158	158	158
query27	2831	275	283	275
query28	7525	2062	2023	2023
query29	804	408	411	408
query30	311	163	153	153
query31	981	759	757	757
query32	96	57	57	57
query33	755	293	286	286
query34	983	478	474	474
query35	869	776	714	714
query36	1062	943	955	943
query37	151	87	83	83
query38	4036	3840	3914	3840
query39	1478	1390	1385	1385
query40	209	118	113	113
query41	47	47	46	46
query42	113	97	100	97
query43	523	478	482	478
query44	1264	775	748	748
query45	198	167	168	167
query46	1095	760	739	739
query47	1899	1797	1829	1797
query48	362	298	303	298
query49	1104	440	436	436
query50	815	408	408	408
query51	7059	6922	6957	6922
query52	102	85	86	85
query53	258	188	187	187
query54	933	456	456	456
query55	78	74	75	74
query56	270	260	283	260
query57	1207	1143	1058	1058
query58	248	227	243	227
query59	3115	2836	3011	2836
query60	289	265	267	265
query61	105	100	100	100
query62	861	648	676	648
query63	219	183	181	181
query64	5086	705	651	651
query65	3247	3193	3146	3146
query66	1399	351	338	338
query67	15578	15361	15317	15317
query68	3535	590	579	579
query69	399	270	282	270
query70	1122	1131	1053	1053
query71	333	275	270	270
query72	6377	4294	3962	3962
query73	744	330	331	330
query74	9159	8798	8852	8798
query75	3410	2723	2674	2674
query76	1933	1052	974	974
query77	491	327	321	321
query78	9838	9070	9136	9070
query79	1033	550	541	541
query80	695	509	510	509
query81	506	240	239	239
query82	235	143	144	143
query83	170	167	155	155
query84	235	79	79	79
query85	676	298	275	275
query86	324	309	298	298
query87	4503	4184	4308	4184
query88	2910	2350	2328	2328
query89	379	290	295	290
query90	2022	195	192	192
query91	126	99	106	99
query92	64	50	50	50
query93	1038	561	542	542
query94	698	289	294	289
query95	342	268	254	254
query96	590	265	267	265
query97	3224	3069	3048	3048
query98	221	206	201	201
query99	1449	1310	1267	1267
Total cold run time: 284439 ms
Total hot run time: 187543 ms

doris-robot · 2024-09-06T03:35:53Z

ClickBench: Total hot run time: 31.56 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 9da97246649e9f8ace5bca4477961508ceafce77, data reload: false
query1	0.04	0.04	0.04
query2	0.07	0.04	0.04
query3	0.23	0.05	0.05
query4	1.67	0.07	0.08
query5	0.52	0.48	0.50
query6	1.13	0.74	0.73
query7	0.02	0.02	0.01
query8	0.06	0.04	0.04
query9	0.54	0.48	0.48
query10	0.56	0.58	0.55
query11	0.15	0.12	0.12
query12	0.16	0.12	0.12
query13	0.61	0.58	0.58
query14	1.41	1.44	1.38
query15	0.83	0.82	0.82
query16	0.37	0.38	0.38
query17	1.06	1.04	0.98
query18	0.22	0.20	0.21
query19	1.92	1.77	1.86
query20	0.01	0.01	0.01
query21	15.39	0.67	0.66
query22	3.79	7.46	2.21
query23	18.30	1.39	1.19
query24	2.11	0.22	0.22
query25	0.15	0.08	0.08
query26	0.27	0.18	0.17
query27	0.08	0.08	0.07
query28	13.28	1.02	1.00
query29	12.62	3.35	3.34
query30	0.24	0.06	0.06
query31	2.88	0.40	0.39
query32	3.25	0.47	0.48
query33	2.99	3.00	2.97
query34	16.88	4.41	4.37
query35	4.52	4.46	4.43
query36	0.66	0.47	0.49
query37	0.19	0.17	0.16
query38	0.16	0.15	0.14
query39	0.05	0.04	0.04
query40	0.15	0.14	0.13
query41	0.10	0.04	0.04
query42	0.06	0.06	0.06
query43	0.05	0.05	0.04
Total cold run time: 109.75 s
Total hot run time: 31.56 s

github-actions · 2024-09-06T08:53:26Z

PR approved by at least one committer and no changes requested.

github-actions · 2024-09-06T08:53:29Z

PR approved by anyone and no changes requested.

…n nereids

zy-kkk · 2024-09-10T07:00:21Z

run buildall

doris-robot · 2024-09-10T12:53:12Z

TPC-H: Total hot run time: 38380 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 4d4ccbcb19a8f225bc9a18ba3d5811c0fca0cc77, data reload: false
------ Round 1 ----------------------------------
q1	17652	4450	4298	4298
q2	2018	194	194	194
q3	11828	968	1151	968
q4	10501	854	782	782
q5	7732	2873	2823	2823
q6	228	142	138	138
q7	979	627	623	623
q8	9313	2077	2089	2077
q9	7213	6547	6558	6547
q10	6997	2266	2198	2198
q11	461	246	244	244
q12	404	224	230	224
q13	17760	3140	3147	3140
q14	281	235	240	235
q15	535	485	492	485
q16	531	448	445	445
q17	1002	716	693	693
q18	7451	6850	6856	6850
q19	1392	1112	1134	1112
q20	672	329	344	329
q21	3877	3105	2976	2976
q22	1124	1040	999	999
Total cold run time: 109951 ms
Total hot run time: 38380 ms
----- Round 2, with runtime_filter_mode=off -----
q1	4357	4277	4266	4266
q2	394	272	270	270
q3	2906	2652	2705	2652
q4	1907	1640	1646	1640
q5	5684	5737	5791	5737
q6	229	136	137	136
q7	2239	1835	1857	1835
q8	3333	3451	3505	3451
q9	8916	8807	8860	8807
q10	3650	3402	3359	3359
q11	619	518	553	518
q12	818	681	678	678
q13	14733	3316	3297	3297
q14	333	305	283	283
q15	540	507	487	487
q16	557	515	531	515
q17	1859	1576	1535	1535
q18	8353	8023	7936	7936
q19	1746	1508	1625	1508
q20	2139	1938	1929	1929
q21	5786	5452	5313	5313
q22	1121	1047	1079	1047
Total cold run time: 72219 ms
Total hot run time: 57199 ms

morningman

LGTM

github-actions · 2024-09-11T02:00:02Z

PR approved by at least one committer and no changes requested.

… in nereids (apache#39813)

… insert stmt in nereids (#40902) pick (#39813)

… in nereids (apache#39813)

… stmt in nereids (#41511) pick (#39813)

… in nereids (apache#39813)

zy-kkk force-pushed the jdbc_sink_new_plan branch from 532f354 to 559b20cCompare August 27, 2024 07:20

zy-kkk marked this pull request as ready for review September 5, 2024 02:36

morrySnow reviewed Sep 5, 2024
View reviewed changes

zy-kkk force-pushed the jdbc_sink_new_plan branch from 559b20c to 9da9724Compare September 6, 2024 02:58

zy-kkk changed the title ~~[improvement](jdbc catalog) support jdbc external table insert stmt in nereids~~[improvement](jdbc catalog) support jdbc external catalog insert stmt in nereidsSep 6, 2024

morrySnow previously approved these changes Sep 6, 2024
View reviewed changes

github-actionsBot added approved Indicates a PR has been approved by one committer. reviewed labels Sep 6, 2024

morrySnow added the dev/2.1.x label Sep 6, 2024

zy-kkk added 3 commits September 10, 2024 11:39

[improvement](jdbc catalog) support jdbc external table insert stmt i…
6c8e6f6
…n nereids

fix comment
1bedc84

add getJdbcColumnToOutput
4d4ccbc

zy-kkk dismissed morrySnow’s stale review via 4d4ccbcSeptember 10, 2024 07:00

zy-kkk force-pushed the jdbc_sink_new_plan branch from 9da9724 to 4d4ccbcCompare September 10, 2024 07:00

github-actionsBot removed the approved Indicates a PR has been approved by one committer. label Sep 10, 2024

morningman approved these changes Sep 11, 2024
View reviewed changes

morningman added the dev/3.0.x label Sep 11, 2024

github-actionsBot added the approved Indicates a PR has been approved by one committer. label Sep 11, 2024

morrySnow approved these changes Sep 11, 2024
View reviewed changes

morrySnow merged commit 3be6ed4 into apache:masterSep 11, 2024

zy-kkk deleted the jdbc_sink_new_plan branch September 11, 2024 06:17

zy-kkk added a commit to zy-kkk/doris that referenced this pull request Sep 18, 2024
[improvement](jdbc catalog) support jdbc external catalog insert stmt…
6d4b9b2
… in nereids (apache#39813)

zy-kkk mentioned this pull request Sep 18, 2024
[branch-2.1][improvement](jdbc catalog) support jdbc external catalog insert stmt in nereids #40902
Merged

yiguolei pushed a commit that referenced this pull request Sep 18, 2024
[branch-2.1][improvement](jdbc catalog) support jdbc external catalog…
f3a42fa
… insert stmt in nereids (#40902) pick (#39813)

yiguolei added dev/2.1.7-merged and removed dev/2.1.x labels Sep 18, 2024

zy-kkk added a commit to zy-kkk/doris that referenced this pull request Sep 30, 2024
[improvement](jdbc catalog) support jdbc external catalog insert stmt…
7bb35e1
… in nereids (apache#39813)

zy-kkk mentioned this pull request Sep 30, 2024
[improvement](jdbc catalog) support jdbc external catalog insert stmt in nereids #41511
Merged

zy-kkk added a commit that referenced this pull request Sep 30, 2024
[3.0][improvement](jdbc catalog) support jdbc external catalog insert…
04e842d
… stmt in nereids (#41511) pick (#39813)

zy-kkk added dev/3.0.2-merged and removed dev/3.0.x labels Sep 30, 2024

HappenLee pushed a commit to HappenLee/incubator-doris that referenced this pull request Apr 24, 2026
[improvement](jdbc catalog) support jdbc external catalog insert stmt…
480e2ea
… in nereids (apache#39813)

		return new UnboundJdbcTableSink<>(nameParts, colNames, hints, partitions,
		dmlCommandType, groupExpression, Optional.empty(), children.get(0));

Uh oh!

Conversation

zy-kkk commented Aug 22, 2024

Uh oh!

doris-robot commented Aug 22, 2024

Uh oh!

zy-kkk commented Aug 27, 2024

Uh oh!

doris-robot commented Aug 27, 2024

Uh oh!

doris-robot commented Aug 27, 2024

Uh oh!

doris-robot commented Aug 27, 2024

Uh oh!

morrySnowSep 5, 2024

Choose a reason for hiding this comment

Uh oh!

morrySnowSep 5, 2024

Choose a reason for hiding this comment

Uh oh!

morrySnowSep 5, 2024

Choose a reason for hiding this comment

Uh oh!

zy-kkk commented Sep 6, 2024

Uh oh!

doris-robot commented Sep 6, 2024

Uh oh!

doris-robot commented Sep 6, 2024

Uh oh!

doris-robot commented Sep 6, 2024

Uh oh!

github-actionsBot commented Sep 6, 2024

Uh oh!

github-actionsBot commented Sep 6, 2024

Uh oh!

zy-kkk commented Sep 10, 2024

Uh oh!

doris-robot commented Sep 10, 2024

Uh oh!

morningman left a comment

Choose a reason for hiding this comment

Uh oh!

github-actionsBot commented Sep 11, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants