[improvement](sqlserver catalog) Configurable whether to use encrypt when connecting to SQL Server using the catalog by zy-kkk · Pull Request #36659 · apache/doris

zy-kkk · 2024-06-21T06:40:59Z

In previous versions, we used druid as the default JDBC connection pool, which can use custom decryption to parse the certificate when SQL Server encryption is turned on. However, in the new version, after changing HikariCP as the default connection pool, the SQLServer certificate cannot be parsed, so encryption needs to be turned off for normal use. Therefore, a parameter is added to decide whether to disable SQLServer encryption. It is not disabled by default.

…ion when connecting to SQL Server using the catalog

doris-robot · 2024-06-21T06:41:03Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

zy-kkk · 2024-06-21T06:41:18Z

run buildall

doris-robot · 2024-06-21T07:19:39Z

TPC-H: Total hot run time: 39487 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit d93bdf6dd099aae1dee051914a7ec76024b433aa, data reload: false
------ Round 1 ----------------------------------
q1	17635	4370	4226	4226
q2	2033	193	193	193
q3	10452	1102	1246	1102
q4	10204	757	729	729
q5	7457	2675	2568	2568
q6	221	135	137	135
q7	951	616	586	586
q8	9217	2086	2049	2049
q9	9520	6504	6452	6452
q10	8892	3687	3744	3687
q11	477	251	233	233
q12	493	235	232	232
q13	17760	2965	2946	2946
q14	260	224	216	216
q15	511	489	481	481
q16	503	376	377	376
q17	954	710	665	665
q18	7941	7381	7261	7261
q19	7758	1538	1493	1493
q20	657	337	337	337
q21	4876	3186	3852	3186
q22	408	376	334	334
Total cold run time: 119180 ms
Total hot run time: 39487 ms
----- Round 2, with runtime_filter_mode=off -----
q1	4371	4305	4246	4246
q2	362	272	270	270
q3	3084	2893	2915	2893
q4	1945	1741	1711	1711
q5	5538	5460	5424	5424
q6	222	128	126	126
q7	2253	1843	1813	1813
q8	3274	3426	3363	3363
q9	8654	8733	8728	8728
q10	4125	3826	3802	3802
q11	604	509	510	509
q12	821	648	612	612
q13	15991	3222	3148	3148
q14	314	278	267	267
q15	524	485	477	477
q16	499	454	446	446
q17	1810	1517	1496	1496
q18	8026	7760	7711	7711
q19	1765	1667	1627	1627
q20	3155	1894	1855	1855
q21	5094	4942	4830	4830
q22	653	545	594	545
Total cold run time: 73084 ms
Total hot run time: 55899 ms

doris-robot · 2024-06-21T07:30:33Z

TPC-DS: Total hot run time: 174138 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit d93bdf6dd099aae1dee051914a7ec76024b433aa, data reload: false
query1	916	386	379	379
query2	6458	2457	2305	2305
query3	6625	210	217	210
query4	20260	17421	17224	17224
query5	3645	494	490	490
query6	247	164	160	160
query7	4585	291	283	283
query8	319	295	298	295
query9	8565	2435	2408	2408
query10	584	294	278	278
query11	10417	10019	9984	9984
query12	124	85	85	85
query13	1643	370	362	362
query14	10047	7658	7729	7658
query15	248	199	198	198
query16	7744	283	273	273
query17	1861	549	530	530
query18	2024	287	280	280
query19	241	166	165	165
query20	107	82	87	82
query21	212	136	132	132
query22	4650	4704	4455	4455
query23	33996	33635	33801	33635
query24	10548	2796	2779	2779
query25	561	354	360	354
query26	705	146	149	146
query27	2219	312	313	312
query28	5903	2072	2057	2057
query29	856	610	596	596
query30	210	151	149	149
query31	938	739	743	739
query32	85	57	58	57
query33	672	308	293	293
query34	865	479	459	459
query35	757	629	638	629
query36	1089	917	925	917
query37	139	76	72	72
query38	2893	2785	2766	2766
query39	858	799	840	799
query40	205	127	130	127
query41	56	54	53	53
query42	125	99	102	99
query43	565	537	582	537
query44	1093	732	739	732
query45	198	166	170	166
query46	1065	700	708	700
query47	1863	1760	1752	1752
query48	365	295	299	295
query49	849	460	428	428
query50	774	398	392	392
query51	6928	6778	6775	6775
query52	108	87	97	87
query53	360	295	285	285
query54	892	445	441	441
query55	75	73	72	72
query56	293	263	259	259
query57	1123	1017	1036	1017
query58	266	252	254	252
query59	3383	3160	3268	3160
query60	292	268	276	268
query61	91	97	123	97
query62	624	453	444	444
query63	326	287	287	287
query64	8480	2230	1775	1775
query65	3164	3078	3106	3078
query66	742	329	322	322
query67	15604	15328	14958	14958
query68	4522	577	527	527
query69	529	469	375	375
query70	1119	1065	1171	1065
query71	397	276	282	276
query72	7104	5112	5050	5050
query73	736	321	318	318
query74	5865	5510	5554	5510
query75	3400	2669	2662	2662
query76	2575	933	918	918
query77	426	298	299	298
query78	10433	9783	9692	9692
query79	2125	506	514	506
query80	997	476	468	468
query81	593	220	220	220
query82	795	105	103	103
query83	259	168	168	168
query84	235	86	91	86
query85	1744	283	278	278
query86	491	342	338	338
query87	3261	3074	3074	3074
query88	3954	2366	2352	2352
query89	484	373	379	373
query90	1743	186	189	186
query91	127	100	101	100
query92	63	49	49	49
query93	2610	491	481	481
query94	1081	254	183	183
query95	414	319	309	309
query96	587	270	261	261
query97	3214	3019	3015	3015
query98	220	202	196	196
query99	1211	848	863	848
Total cold run time: 268039 ms
Total hot run time: 174138 ms

doris-robot · 2024-06-21T07:36:06Z

ClickBench: Total hot run time: 31.52 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit d93bdf6dd099aae1dee051914a7ec76024b433aa, data reload: false
query1	0.05	0.04	0.04
query2	0.08	0.04	0.04
query3	0.22	0.04	0.06
query4	1.67	0.10	0.07
query5	0.48	0.49	0.50
query6	1.13	0.73	0.72
query7	0.02	0.02	0.02
query8	0.05	0.04	0.05
query9	0.54	0.48	0.50
query10	0.55	0.56	0.53
query11	0.15	0.12	0.11
query12	0.16	0.13	0.13
query13	0.60	0.60	0.61
query14	0.79	0.78	0.77
query15	0.86	0.82	0.81
query16	0.36	0.36	0.37
query17	0.99	0.99	1.01
query18	0.23	0.25	0.25
query19	1.78	1.72	1.73
query20	0.02	0.01	0.01
query21	15.40	0.65	0.66
query22	4.23	6.80	2.62
query23	18.34	1.46	1.31
query24	2.19	0.22	0.22
query25	0.14	0.09	0.08
query26	0.27	0.18	0.18
query27	0.08	0.09	0.08
query28	13.18	1.02	1.00
query29	12.55	3.33	3.33
query30	0.26	0.06	0.08
query31	2.84	0.39	0.39
query32	3.28	0.47	0.48
query33	2.95	2.94	2.92
query34	17.16	4.42	4.42
query35	4.45	4.50	4.47
query36	0.62	0.46	0.46
query37	0.19	0.16	0.16
query38	0.16	0.15	0.15
query39	0.04	0.04	0.04
query40	0.18	0.15	0.15
query41	0.10	0.06	0.05
query42	0.06	0.06	0.05
query43	0.05	0.04	0.04
Total cold run time: 109.45 s
Total hot run time: 31.52 s

morningman

LGTM

github-actions · 2024-06-21T15:06:48Z

PR approved by at least one committer and no changes requested.

github-actions · 2024-06-21T15:06:51Z

PR approved by anyone and no changes requested.

dataroaring

LGTM

zy-kkk · 2024-06-27T06:57:09Z

run external

zy-kkk · 2024-06-27T07:21:25Z

run buildall

doris-robot · 2024-06-27T07:58:15Z

TPC-H: Total hot run time: 39409 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit d93bdf6dd099aae1dee051914a7ec76024b433aa, data reload: false
------ Round 1 ----------------------------------
q1	17611	4709	4284	4284
q2	2025	189	194	189
q3	10474	1180	1179	1179
q4	10186	818	717	717
q5	7491	2712	2609	2609
q6	212	133	132	132
q7	934	590	600	590
q8	9227	2060	2048	2048
q9	8989	6447	6523	6447
q10	8977	3709	3685	3685
q11	441	235	232	232
q12	487	230	220	220
q13	18508	2984	2974	2974
q14	257	218	221	218
q15	513	461	465	461
q16	527	377	366	366
q17	959	625	654	625
q18	8002	7361	7296	7296
q19	4688	1436	1502	1436
q20	648	314	311	311
q21	4870	3049	3846	3049
q22	404	346	341	341
Total cold run time: 116430 ms
Total hot run time: 39409 ms
----- Round 2, with runtime_filter_mode=off -----
q1	4330	4239	4253	4239
q2	356	261	260	260
q3	2942	2787	2886	2787
q4	1955	1787	1671	1671
q5	5639	5466	5509	5466
q6	216	131	124	124
q7	2202	1862	1874	1862
q8	3224	3379	3423	3379
q9	8667	8634	8810	8634
q10	4079	3869	3657	3657
q11	605	487	505	487
q12	812	627	644	627
q13	15888	3211	3170	3170
q14	309	300	286	286
q15	506	490	488	488
q16	482	433	428	428
q17	1838	1517	1511	1511
q18	8076	7762	7914	7762
q19	1782	1634	1515	1515
q20	3026	1946	1823	1823
q21	5077	4939	4872	4872
q22	830	566	551	551
Total cold run time: 72841 ms
Total hot run time: 55599 ms

doris-robot · 2024-06-27T08:09:18Z

TPC-DS: Total hot run time: 174004 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit d93bdf6dd099aae1dee051914a7ec76024b433aa, data reload: false
query1	917	383	377	377
query2	6473	2452	2371	2371
query3	6629	214	219	214
query4	19375	17503	17478	17478
query5	3692	470	461	461
query6	265	175	184	175
query7	4599	286	303	286
query8	316	274	285	274
query9	8587	2331	2311	2311
query10	585	282	278	278
query11	10522	10148	10172	10148
query12	111	88	84	84
query13	1636	363	359	359
query14	9577	7544	7653	7544
query15	233	187	189	187
query16	7624	261	263	261
query17	1493	532	541	532
query18	1952	282	299	282
query19	195	150	148	148
query20	90	80	83	80
query21	218	126	122	122
query22	4439	4174	3963	3963
query23	33539	33646	33374	33374
query24	11364	2942	2861	2861
query25	631	380	359	359
query26	1239	158	160	158
query27	3137	323	331	323
query28	7717	2082	2050	2050
query29	892	625	605	605
query30	265	162	163	162
query31	976	741	771	741
query32	89	53	58	53
query33	738	294	299	294
query34	1079	487	489	487
query35	768	670	644	644
query36	1121	973	948	948
query37	153	68	72	68
query38	2954	2882	2802	2802
query39	858	805	822	805
query40	208	123	119	119
query41	57	50	54	50
query42	118	99	100	99
query43	635	529	580	529
query44	1217	735	729	729
query45	188	164	166	164
query46	1069	701	696	696
query47	1875	1799	1833	1799
query48	375	310	306	306
query49	830	392	407	392
query50	766	391	382	382
query51	6933	6789	6736	6736
query52	103	98	91	91
query53	352	293	284	284
query54	878	444	438	438
query55	75	72	71	71
query56	279	281	253	253
query57	1135	1029	1094	1029
query58	237	250	239	239
query59	3470	3182	3378	3182
query60	290	268	270	268
query61	94	89	91	89
query62	628	440	448	440
query63	310	284	288	284
query64	8843	2249	1868	1868
query65	3164	3067	3079	3067
query66	772	332	335	332
query67	15623	15441	14818	14818
query68	4524	539	531	531
query69	618	451	396	396
query70	1183	1149	1162	1149
query71	453	273	274	273
query72	7215	5164	5561	5164
query73	751	324	321	321
query74	5927	5447	5504	5447
query75	3419	2664	2707	2664
query76	2840	901	948	901
query77	634	301	295	295
query78	10445	10107	9763	9763
query79	2546	527	506	506
query80	1639	472	459	459
query81	577	223	219	219
query82	844	105	104	104
query83	353	172	172	172
query84	241	84	85	84
query85	1885	282	285	282
query86	474	284	302	284
query87	3252	3174	3072	3072
query88	4245	2472	2439	2439
query89	476	385	382	382
query90	1721	192	196	192
query91	131	102	98	98
query92	57	51	51	51
query93	3009	498	499	498
query94	1112	196	193	193
query95	411	327	326	326
query96	611	272	274	272
query97	3240	3107	3023	3023
query98	215	204	199	199
query99	1148	839	822	822
Total cold run time: 273768 ms
Total hot run time: 174004 ms

doris-robot · 2024-06-27T08:14:49Z

ClickBench: Total hot run time: 31.05 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit d93bdf6dd099aae1dee051914a7ec76024b433aa, data reload: false
query1	0.04	0.03	0.03
query2	0.08	0.04	0.04
query3	0.23	0.06	0.06
query4	1.65	0.08	0.08
query5	0.51	0.50	0.50
query6	1.13	0.73	0.72
query7	0.02	0.01	0.02
query8	0.05	0.04	0.05
query9	0.55	0.50	0.48
query10	0.55	0.55	0.55
query11	0.16	0.11	0.11
query12	0.16	0.12	0.13
query13	0.59	0.59	0.62
query14	0.77	0.79	0.77
query15	0.84	0.81	0.81
query16	0.37	0.37	0.37
query17	1.05	1.01	0.99
query18	0.21	0.24	0.27
query19	1.81	1.76	1.77
query20	0.01	0.01	0.01
query21	15.46	0.75	0.66
query22	4.20	7.28	2.37
query23	18.31	1.29	1.20
query24	2.17	0.21	0.21
query25	0.13	0.09	0.08
query26	0.26	0.18	0.17
query27	0.08	0.09	0.09
query28	13.24	1.01	1.00
query29	12.58	3.29	3.25
query30	0.25	0.07	0.05
query31	2.87	0.39	0.39
query32	3.27	0.46	0.48
query33	2.88	2.91	2.84
query34	17.14	4.43	4.45
query35	4.48	4.55	4.50
query36	0.64	0.46	0.46
query37	0.19	0.17	0.15
query38	0.16	0.15	0.15
query39	0.05	0.04	0.04
query40	0.18	0.14	0.15
query41	0.09	0.05	0.05
query42	0.05	0.04	0.05
query43	0.05	0.05	0.04
Total cold run time: 109.51 s
Total hot run time: 31.05 s

…when connecting to SQL Server using the catalog (apache#36659) In previous versions, we used druid as the default JDBC connection pool, which can use custom decryption to parse the certificate when SQL Server encryption is turned on. However, in the new version, after changing HikariCP as the default connection pool, the SQLServer certificate cannot be parsed, so encryption needs to be turned off for normal use. Therefore, a parameter is added to decide whether to disable SQLServer encryption. It is not disabled by default.

…when connecting to SQL Server using the catalog (#36659) In previous versions, we used druid as the default JDBC connection pool, which can use custom decryption to parse the certificate when SQL Server encryption is turned on. However, in the new version, after changing HikariCP as the default connection pool, the SQLServer certificate cannot be parsed, so encryption needs to be turned off for normal use. Therefore, a parameter is added to decide whether to disable SQLServer encryption. It is not disabled by default.

…_encrypt` to `force_sqlserver_jdbc_encrypt_false` (#37015) Front #36659 Renamed the configuration parameter `disable_jdbc_sqlserver_encrypt` to `force_sqlserver_jdbc_encrypt_false` to make its purpose clearer and more explicit. This new name better indicates that the parameter forces the JDBC URL to set `encrypt=false` when enabled.

… use encryption when connecting to SQL Server using the catalog (#36940) pick #36659 pick #37015

…use encrypt when connecting to SQL Server using the catalog (#36971) pick (#36659) pick #37015 In previous versions, we used druid as the default JDBC connection pool, which can use custom decryption to parse the certificate when SQL Server encryption is turned on. However, in the new version, after changing HikariCP as the default connection pool, the SQLServer certificate cannot be parsed, so encryption needs to be turned off for normal use. Therefore, a parameter is added to decide whether to disable SQLServer encryption. It is not disabled by default.

…_encrypt` to `force_sqlserver_jdbc_encrypt_false` (#37015) Front #36659 Renamed the configuration parameter `disable_jdbc_sqlserver_encrypt` to `force_sqlserver_jdbc_encrypt_false` to make its purpose clearer and more explicit. This new name better indicates that the parameter forces the JDBC URL to set `encrypt=false` when enabled.

… use encryption when connecting to SQL Server using the catalog (apache#36940) pick apache#36659 pick apache#37015

…when connecting to SQL Server using the catalog (apache#36659) In previous versions, we used druid as the default JDBC connection pool, which can use custom decryption to parse the certificate when SQL Server encryption is turned on. However, in the new version, after changing HikariCP as the default connection pool, the SQLServer certificate cannot be parsed, so encryption needs to be turned off for normal use. Therefore, a parameter is added to decide whether to disable SQLServer encryption. It is not disabled by default.

…_encrypt` to `force_sqlserver_jdbc_encrypt_false` (apache#37015) Front apache#36659 Renamed the configuration parameter `disable_jdbc_sqlserver_encrypt` to `force_sqlserver_jdbc_encrypt_false` to make its purpose clearer and more explicit. This new name better indicates that the parameter forces the JDBC URL to set `encrypt=false` when enabled.

[improvement](sqlserver catalog) Can configure whether to use encrypt…
d93bdf6
…ion when connecting to SQL Server using the catalog

zy-kkk added dev/2.0.x dev/2.1.x labels Jun 21, 2024

morningman approved these changes Jun 21, 2024
View reviewed changes

github-actionsBot added the approved Indicates a PR has been approved by one committer. label Jun 21, 2024

github-actionsBot added the reviewed label Jun 21, 2024

wm1581066 added the usercase Important user case type label label Jun 26, 2024

dataroaring approved these changes Jun 27, 2024
View reviewed changes

zy-kkk merged commit dd77c84 into apache:masterJun 27, 2024

zy-kkk deleted the sqlserver_encrypt_false branch June 27, 2024 09:55

zy-kkk mentioned this pull request Jun 27, 2024
[branch-2.0][improvement](sqlserver catalog) Can configure whether to use encryption when connecting to SQL Server using the catalog #36940
Merged

zy-kkk mentioned this pull request Jun 28, 2024
[branch-2.1][improvement](sqlserver catalog) Configurable whether to use encrypt when connecting to SQL Server using the catalog #36971
Merged

zy-kkk mentioned this pull request Jun 28, 2024
[improvement](jdbc catalog)Rename config from disable_jdbc_sqlserver_encrypt to force_sqlserver_jdbc_encrypt_false#37015
Merged

morningman pushed a commit that referenced this pull request Jul 1, 2024
[branch-2.0][improvement](sqlserver catalog) Can configure whether to…
4c4f0be
… use encryption when connecting to SQL Server using the catalog (#36940) pick #36659 pick #37015

morningman removed the dev/2.0.x label Jul 1, 2024

morningman added the dev/2.0.13-merged label Jul 1, 2024

wm1581066 added dev/2.1.5-merged and removed dev/2.1.x labels Jul 2, 2024

dataroaring added the dev/3.0.0-merged label Aug 5, 2024

mongo360 pushed a commit to mongo360/doris that referenced this pull request Aug 16, 2024
[branch-2.0][improvement](sqlserver catalog) Can configure whether to…
e83c6ac
… use encryption when connecting to SQL Server using the catalog (apache#36940) pick apache#36659 pick apache#37015

Uh oh!

Conversation

zy-kkk commented Jun 21, 2024

Uh oh!

doris-robot commented Jun 21, 2024

Uh oh!

zy-kkk commented Jun 21, 2024

Uh oh!

doris-robot commented Jun 21, 2024

Uh oh!

doris-robot commented Jun 21, 2024

Uh oh!

doris-robot commented Jun 21, 2024

Uh oh!

morningman left a comment

Choose a reason for hiding this comment

Uh oh!

github-actionsBot commented Jun 21, 2024

Uh oh!

github-actionsBot commented Jun 21, 2024

Uh oh!

dataroaring left a comment

Choose a reason for hiding this comment

Uh oh!

zy-kkk commented Jun 27, 2024

Uh oh!

zy-kkk commented Jun 27, 2024

Uh oh!

doris-robot commented Jun 27, 2024

Uh oh!

doris-robot commented Jun 27, 2024

Uh oh!

doris-robot commented Jun 27, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants