[fix](arrow-flight-sql) Fix exceed user property max connection cause `Reach limit of connections` by xinyiZzz · Pull Request #39127 · apache/doris

xinyiZzz · 2024-08-08T14:37:55Z

Limit the number of arrow flight connections for a single user to less than the user property max_user_connections / 2, default 50.

doris-robot · 2024-08-08T14:38:00Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

xinyiZzz · 2024-08-08T14:38:01Z

run buildall

doris-robot · 2024-08-08T15:16:24Z

TPC-H: Total hot run time: 38916 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 2f4a8006cb8a208177c442e7b08b851dccaa6d60, data reload: false
------ Round 1 ----------------------------------
q1	17608	4310	4298	4298
q2	2028	175	192	175
q3	10460	1207	1098	1098
q4	10137	721	713	713
q5	7500	2504	2464	2464
q6	223	135	136	135
q7	958	597	594	594
q8	9217	1873	1910	1873
q9	8734	6490	6569	6490
q10	7045	2159	2265	2159
q11	447	239	240	239
q12	496	220	225	220
q13	18788	2969	2955	2955
q14	284	235	244	235
q15	523	491	466	466
q16	494	399	379	379
q17	960	679	707	679
q18	7946	7433	7361	7361
q19	3643	972	1076	972
q20	687	330	323	323
q21	5306	4095	4598	4095
q22	1095	994	993	993
Total cold run time: 114579 ms
Total hot run time: 38916 ms
----- Round 2, with runtime_filter_mode=off -----
q1	4535	4274	4283	4274
q2	368	284	268	268
q3	2851	2609	2630	2609
q4	1972	1699	1717	1699
q5	5518	5602	5472	5472
q6	225	131	132	131
q7	2131	1730	1728	1728
q8	3345	3435	3396	3396
q9	8806	8708	8832	8708
q10	3518	3207	3301	3207
q11	630	511	520	511
q12	804	617	616	616
q13	17335	3151	3161	3151
q14	320	288	306	288
q15	538	491	491	491
q16	491	447	437	437
q17	1853	1536	1513	1513
q18	8186	7931	7828	7828
q19	1713	1601	1670	1601
q20	2096	1890	1929	1890
q21	12744	5286	5334	5286
q22	1133	1021	1023	1021
Total cold run time: 81112 ms
Total hot run time: 56125 ms

doris-robot · 2024-08-08T15:28:56Z

TPC-DS: Total hot run time: 203041 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 2f4a8006cb8a208177c442e7b08b851dccaa6d60, data reload: false
query1	951	418	396	396
query2	6427	1894	1775	1775
query3	6647	206	215	206
query4	33992	23170	23070	23070
query5	3624	502	496	496
query6	287	174	170	170
query7	4566	287	284	284
query8	242	203	198	198
query9	8504	2370	2351	2351
query10	988	882	911	882
query11	17756	15030	15012	15012
query12	134	100	95	95
query13	1627	377	363	363
query14	10635	8092	7988	7988
query15	395	342	324	324
query16	7484	452	507	452
query17	1524	622	579	579
query18	1368	405	387	387
query19	280	238	185	185
query20	118	111	115	111
query21	232	111	106	106
query22	4685	4789	4426	4426
query23	34185	33236	33657	33236
query24	8347	2998	2932	2932
query25	565	378	357	357
query26	693	149	149	149
query27	2154	275	285	275
query28	5520	2006	1978	1978
query29	707	406	400	400
query30	252	148	142	142
query31	994	734	755	734
query32	95	58	53	53
query33	535	287	281	281
query34	865	468	478	468
query35	966	880	822	822
query36	1058	933	913	913
query37	132	81	80	80
query38	4269	4186	4079	4079
query39	1413	1353	1376	1353
query40	205	122	118	118
query41	47	47	47	47
query42	115	95	133	95
query43	484	441	452	441
query44	1112	732	743	732
query45	381	382	379	379
query46	1105	788	791	788
query47	1883	1748	1785	1748
query48	381	304	299	299
query49	824	417	423	417
query50	809	406	408	406
query51	6763	6776	6614	6614
query52	99	92	86	86
query53	254	177	179	177
query54	566	440	455	440
query55	76	75	80	75
query56	280	256	245	245
query57	1141	1109	1101	1101
query58	219	220	227	220
query59	2973	2581	2654	2581
query60	327	255	258	255
query61	97	92	97	92
query62	798	646	634	634
query63	205	177	176	176
query64	9240	2410	1922	1922
query65	3222	3114	3128	3114
query66	745	373	321	321
query67	15280	14886	14839	14839
query68	4486	528	542	528
query69	464	373	383	373
query70	1148	1094	1049	1049
query71	424	275	268	268
query72	18388	16491	16610	16491
query73	769	325	326	325
query74	9211	8738	8856	8738
query75	3425	2677	2688	2677
query76	2521	1001	1044	1001
query77	650	311	313	311
query78	9953	9103	9648	9103
query79	2672	518	519	518
query80	1687	480	482	480
query81	603	226	231	226
query82	826	137	134	134
query83	303	145	156	145
query84	265	83	77	77
query85	1705	332	351	332
query86	466	294	288	288
query87	4589	4535	4521	4521
query88	4138	2594	2518	2518
query89	393	284	282	282
query90	1779	274	193	193
query91	151	116	119	116
query92	61	48	50	48
query93	2617	530	525	525
query94	907	295	287	287
query95	351	260	258	258
query96	609	278	278	278
query97	3227	3064	3046	3046
query98	219	203	196	196
query99	1536	1256	1239	1239
Total cold run time: 305390 ms
Total hot run time: 203041 ms

doris-robot · 2024-08-08T15:34:28Z

ClickBench: Total hot run time: 30.7 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 2f4a8006cb8a208177c442e7b08b851dccaa6d60, data reload: false
query1	0.05	0.04	0.04
query2	0.08	0.04	0.04
query3	0.23	0.05	0.05
query4	1.68	0.09	0.09
query5	0.53	0.48	0.49
query6	1.14	0.73	0.73
query7	0.02	0.01	0.01
query8	0.06	0.05	0.05
query9	0.55	0.49	0.50
query10	0.54	0.55	0.56
query11	0.16	0.12	0.11
query12	0.16	0.12	0.12
query13	0.58	0.60	0.59
query14	0.77	0.80	0.77
query15	0.89	0.83	0.81
query16	0.37	0.38	0.37
query17	1.04	1.04	0.99
query18	0.23	0.22	0.21
query19	1.92	1.70	1.71
query20	0.01	0.01	0.01
query21	15.39	0.74	0.66
query22	4.04	6.86	2.11
query23	18.27	1.35	1.18
query24	2.16	0.23	0.21
query25	0.17	0.08	0.07
query26	0.29	0.21	0.20
query27	0.45	0.22	0.22
query28	13.26	1.01	1.00
query29	12.68	3.27	3.25
query30	0.23	0.05	0.05
query31	2.88	0.38	0.39
query32	3.30	0.47	0.46
query33	2.84	2.91	2.98
query34	16.96	4.32	4.35
query35	4.37	4.41	4.45
query36	0.64	0.47	0.47
query37	0.18	0.15	0.16
query38	0.16	0.15	0.15
query39	0.04	0.03	0.04
query40	0.16	0.12	0.12
query41	0.09	0.05	0.05
query42	0.06	0.05	0.05
query43	0.04	0.04	0.04
Total cold run time: 109.67 s
Total hot run time: 30.7 s

github-actions · 2024-08-08T20:15:35Z

PR approved by at least one committer and no changes requested.

github-actions · 2024-08-08T20:15:36Z

PR approved by anyone and no changes requested.

xinyiZzz · 2024-08-13T12:26:06Z

run buildall

doris-robot · 2024-08-13T13:05:02Z

TPC-H: Total hot run time: 39997 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 59931e2432da428e8b2b2464d9417199ac67091b, data reload: false
------ Round 1 ----------------------------------
q1	17614	4368	4281	4281
q2	2021	181	195	181
q3	11665	1057	1033	1033
q4	10515	679	669	669
q5	7727	2797	2824	2797
q6	226	144	144	144
q7	952	615	604	604
q8	9392	2038	2057	2038
q9	8763	6546	6591	6546
q10	7054	2260	2163	2163
q11	530	264	262	262
q12	393	225	227	225
q13	17883	3030	3049	3030
q14	271	236	245	236
q15	517	490	504	490
q16	516	396	386	386
q17	1024	705	731	705
q18	7940	7562	7438	7438
q19	7493	1098	1070	1070
q20	710	324	323	323
q21	5980	4409	4604	4409
q22	1111	967	1007	967
Total cold run time: 120297 ms
Total hot run time: 39997 ms
----- Round 2, with runtime_filter_mode=off -----
q1	4981	4244	4292	4244
q2	380	276	276	276
q3	3046	2850	2774	2774
q4	2067	1734	1741	1734
q5	5699	5621	5787	5621
q6	232	135	141	135
q7	2237	1794	1894	1794
q8	3393	3557	3515	3515
q9	9191	9032	8861	8861
q10	3631	3426	3365	3365
q11	622	529	521	521
q12	887	683	707	683
q13	16295	3196	3178	3178
q14	318	295	291	291
q15	537	499	520	499
q16	485	448	464	448
q17	1834	1524	1522	1522
q18	8422	8102	7976	7976
q19	1770	1626	1612	1612
q20	2314	1889	1874	1874
q21	9432	5253	5371	5253
q22	1127	1039	1039	1039
Total cold run time: 78900 ms
Total hot run time: 57215 ms

xinyiZzz · 2024-08-21T14:50:36Z

run buildall

doris-robot · 2024-08-21T15:16:46Z

TPC-H: Total hot run time: 39104 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit a0ba97e67a652ca8a78ea8c7b3efb0bcb5ebe490, data reload: false
------ Round 1 ----------------------------------
q1	17937	6060	4366	4366
q2	2063	217	213	213
q3	10503	1166	1113	1113
q4	10364	764	835	764
q5	7777	2913	2925	2913
q6	269	161	163	161
q7	1026	687	668	668
q8	9368	2174	2151	2151
q9	7202	6627	6574	6574
q10	7084	2259	2317	2259
q11	481	294	272	272
q12	428	252	256	252
q13	17799	3128	3025	3025
q14	307	264	255	255
q15	567	505	527	505
q16	517	410	418	410
q17	1017	748	740	740
q18	7546	6963	6981	6963
q19	1444	1044	1129	1044
q20	723	354	376	354
q21	3932	3045	3060	3045
q22	1172	1066	1057	1057
Total cold run time: 109526 ms
Total hot run time: 39104 ms
----- Round 2, with runtime_filter_mode=off -----
q1	4446	4390	4327	4327
q2	437	304	296	296
q3	2900	2715	2742	2715
q4	1954	1730	1690	1690
q5	5452	5471	5445	5445
q6	237	150	147	147
q7	2166	1838	1783	1783
q8	3307	3421	3427	3421
q9	8557	8509	8546	8509
q10	3505	3265	3268	3265
q11	626	534	546	534
q12	828	654	635	635
q13	13473	3057	3097	3057
q14	327	306	309	306
q15	568	534	535	534
q16	522	463	457	457
q17	1826	1540	1505	1505
q18	7839	7538	7493	7493
q19	1759	1576	1746	1576
q20	2081	1868	1924	1868
q21	5584	5430	5389	5389
q22	1154	1086	1061	1061
Total cold run time: 69548 ms
Total hot run time: 56013 ms

doris-robot · 2024-08-21T15:28:56Z

TPC-DS: Total hot run time: 193265 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit a0ba97e67a652ca8a78ea8c7b3efb0bcb5ebe490, data reload: false
query1	973	411	392	392
query2	6724	1984	2056	1984
query3	6661	242	244	242
query4	34308	23203	23151	23151
query5	4392	725	714	714
query6	321	214	212	212
query7	4619	334	331	331
query8	505	437	434	434
query9	8611	2558	2565	2558
query10	514	357	344	344
query11	17930	15083	15247	15083
query12	185	129	128	128
query13	1697	449	432	432
query14	10298	7517	7400	7400
query15	307	192	198	192
query16	8146	536	525	525
query17	1807	613	619	613
query18	2188	348	355	348
query19	359	172	170	170
query20	142	136	141	136
query21	257	150	164	150
query22	4464	4233	4207	4207
query23	34210	33635	33462	33462
query24	11138	2869	2866	2866
query25	665	425	419	419
query26	1384	181	182	181
query27	2746	327	306	306
query28	7319	2121	2119	2119
query29	934	449	441	441
query30	336	188	185	185
query31	1053	850	840	840
query32	128	82	84	82
query33	816	365	350	350
query34	926	526	514	514
query35	916	762	806	762
query36	1114	974	975	974
query37	183	111	115	111
query38	4082	3895	3945	3895
query39	1529	1505	1464	1464
query40	308	158	158	158
query41	147	176	176	176
query42	143	126	133	126
query43	577	548	538	538
query44	1273	818	839	818
query45	246	214	215	214
query46	1173	781	792	781
query47	1996	1926	1850	1850
query48	420	351	346	346
query49	1191	600	596	596
query50	898	477	483	477
query51	7234	7152	7044	7044
query52	120	109	109	109
query53	299	228	225	225
query54	1141	534	518	518
query55	97	91	89	89
query56	345	319	320	319
query57	1251	1155	1175	1155
query58	319	298	376	298
query59	3085	2898	2949	2898
query60	365	338	360	338
query61	152	150	150	150
query62	920	722	698	698
query63	269	228	229	228
query64	6491	2434	1880	1880
query65	3262	3409	3234	3234
query66	1706	680	676	676
query67	15587	15346	15247	15247
query68	4909	617	599	599
query69	482	334	326	326
query70	1171	1084	1119	1084
query71	517	325	322	322
query72	6551	2329	2109	2109
query73	804	365	361	361
query74	9403	9001	8853	8853
query75	3488	2820	2816	2816
query76	2855	1089	1050	1050
query77	633	456	449	449
query78	9786	9134	9446	9134
query79	1684	576	564	564
query80	1137	637	673	637
query81	606	269	259	259
query82	958	160	168	160
query83	329	240	217	217
query84	255	103	102	102
query85	1245	358	351	351
query86	412	333	330	330
query87	4449	4342	4250	4250
query88	3413	2564	2553	2553
query89	431	333	330	330
query90	1807	241	240	240
query91	155	143	131	131
query92	87	75	79	75
query93	1121	568	558	558
query94	937	339	312	312
query95	397	310	311	310
query96	613	286	287	286
query97	3261	3054	3141	3054
query98	249	229	225	225
query99	1770	1315	1343	1315
Total cold run time: 301817 ms
Total hot run time: 193265 ms

doris-robot · 2024-08-21T15:34:29Z

ClickBench: Total hot run time: 31.49 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit a0ba97e67a652ca8a78ea8c7b3efb0bcb5ebe490, data reload: false
query1	0.05	0.04	0.04
query2	0.08	0.04	0.04
query3	0.23	0.06	0.05
query4	1.66	0.08	0.08
query5	0.51	0.49	0.50
query6	1.13	0.74	0.72
query7	0.02	0.01	0.01
query8	0.06	0.05	0.05
query9	0.57	0.50	0.48
query10	0.55	0.56	0.55
query11	0.16	0.12	0.12
query12	0.16	0.12	0.13
query13	0.62	0.60	0.59
query14	0.78	0.79	0.78
query15	0.86	0.83	0.83
query16	0.39	0.38	0.38
query17	1.08	1.06	1.01
query18	0.21	0.21	0.21
query19	1.85	1.89	1.76
query20	0.01	0.01	0.02
query21	15.45	0.68	0.66
query22	3.75	7.46	2.01
query23	18.26	1.47	1.29
query24	2.01	0.24	0.23
query25	0.16	0.09	0.09
query26	0.27	0.19	0.18
query27	0.08	0.08	0.08
query28	13.34	1.05	1.03
query29	12.66	3.44	3.40
query30	0.44	0.25	0.25
query31	2.82	0.41	0.41
query32	3.24	0.50	0.48
query33	2.99	3.00	3.00
query34	16.86	4.41	4.43
query35	4.50	4.45	4.48
query36	0.67	0.48	0.48
query37	0.20	0.18	0.17
query38	0.18	0.16	0.17
query39	0.06	0.05	0.06
query40	0.19	0.16	0.15
query41	0.11	0.07	0.07
query42	0.08	0.07	0.06
query43	0.07	0.06	0.06
Total cold run time: 109.37 s
Total hot run time: 31.49 s

github-actions · 2024-08-22T14:10:55Z

PR approved by at least one committer and no changes requested.

wangbo

LGTM

… `Reach limit of connections` (apache#39127) Limit the number of arrow flight connections for a single user to less than the user property max_user_connections / 2, default 50.

…n cause Reach limit of connections (#39836) pick #39127 pick #39802

… `Reach limit of connections` (#39127) Limit the number of arrow flight connections for a single user to less than the user property max_user_connections / 2, default 50.

… `Reach limit of connections` (apache#39127) Limit the number of arrow flight connections for a single user to less than the user property max_user_connections / 2, default 50.

github-actionsBot added the doing label Aug 8, 2024

yiguolei previously approved these changes Aug 8, 2024
View reviewed changes

yiguolei added the dev/2.1.x label Aug 8, 2024

github-actionsBot added the approved Indicates a PR has been approved by one committer. label Aug 8, 2024

github-actionsBot added the reviewed label Aug 8, 2024

1
f06696c

xinyiZzz force-pushed the 20240730_fix_arrow_connect branch from 59931e2 to f06696cCompare August 21, 2024 12:48

2
a0ba97e

xinyiZzz dismissed yiguolei’s stale review via a0ba97eAugust 21, 2024 14:50

github-actionsBot removed the approved Indicates a PR has been approved by one committer. label Aug 21, 2024

yiguolei approved these changes Aug 22, 2024
View reviewed changes

github-actionsBot added the approved Indicates a PR has been approved by one committer. label Aug 22, 2024

wangbo approved these changes Aug 23, 2024
View reviewed changes

HappenLee merged commit 7a066e6 into apache:masterAug 23, 2024

xinyiZzz mentioned this pull request Aug 23, 2024
[branch-2.1](arrow-flight-sql) Fix exceed user property max connection cause Reach limit of connections #39836
Merged

yiguolei pushed a commit that referenced this pull request Aug 23, 2024
[branch-2.1](arrow-flight-sql) Fix exceed user property max connectio…
e716658
…n cause Reach limit of connections (#39836) pick #39127 pick #39802

yiguolei added dev/2.1.6-merged and removed dev/2.1.x labels Aug 23, 2024

dataroaring added the dev/3.0.3-merged label Oct 12, 2024

gavinchou mentioned this pull request Nov 26, 2024
Release Note 3.0.3 #44522
Closed

Uh oh!

Conversation

xinyiZzz commented Aug 8, 2024

Uh oh!

doris-robot commented Aug 8, 2024

Uh oh!

xinyiZzz commented Aug 8, 2024

Uh oh!

doris-robot commented Aug 8, 2024

Uh oh!

doris-robot commented Aug 8, 2024

Uh oh!

doris-robot commented Aug 8, 2024

Uh oh!

github-actionsBot commented Aug 8, 2024

Uh oh!

github-actionsBot commented Aug 8, 2024

Uh oh!

xinyiZzz commented Aug 13, 2024

Uh oh!

doris-robot commented Aug 13, 2024

Uh oh!

xinyiZzz commented Aug 21, 2024

Uh oh!

doris-robot commented Aug 21, 2024

Uh oh!

doris-robot commented Aug 21, 2024

Uh oh!

doris-robot commented Aug 21, 2024

Uh oh!

github-actionsBot commented Aug 22, 2024

Uh oh!

wangbo left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants