[fix](variables) change column type of @@autocommit to BIGINT by morningman · Pull Request #33282 · apache/doris

morningman · 2024-04-06T12:00:55Z

Proposed changes

Some of mysql connector (eg, dotnet MySQL.Data) rely on variable's column type to make connection.
eg, select @@autocommit should with column type BIGINT, not BIT, otherwise it will throw error like:

System.FormatException: The input string 'True' was not in a correct format.
at System.Number.ThrowFormatException[TChar](ReadOnlySpan`1 value)
at System.Convert.ToInt32(String value)
at MySql.Data.MySqlClient.Driver.LoadCharacterSetsAsync(MySqlConnection connection, Boolean execAsync, CancellationToken cancellationToken)
at MySql.Data.MySqlClient.Driver.ConfigureAsync(MySqlConnection connection, Boolean execAsync, CancellationToken cancellationToken)
at MySql.Data.MySqlClient.MySqlConnection.OpenAsync(Boolean execAsync, CancellationToken cancellationToken)
at MySql.Data.MySqlClient.MySqlConnection.Open()

In this PR, I add a new field of VarAttr: convertBoolToLongMethod, if set, it will convert boolean to long.
And set it for autocommit

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

doris-robot · 2024-04-06T12:01:00Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

morningman · 2024-04-06T12:05:35Z

run buildall

yiguolei · 2024-04-07T01:28:17Z

@@ -3206,14 +3205,14 @@ public void readFromJson(String json) throws IOException {



我觉得这么写太迷惑了，不如直接在加一个新的annotation，比如叫做converter="convertToFromBoolean"
然后在session variable 里定义一下。这样就相当于只对auto commit 这一个变量做一下convert。
否则类似3208行，这是大面上都在做convert，未来很难扩展。

morningman · 2024-04-07T03:44:32Z

run buildall

yiguolei

LGTM

github-actions · 2024-04-07T04:05:36Z

PR approved by at least one committer and no changes requested.

doris-robot · 2024-04-07T04:05:36Z

TPC-H: Total hot run time: 38799 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 55f262b0ff875d871135d2743295bdbffb96ffa0, data reload: false
------ Round 1 ----------------------------------
q1	18555	4208	4164	4164
q2	2484	184	188	184
q3	11209	1272	1335	1272
q4	10529	848	1001	848
q5	8463	2995	2931	2931
q6	218	137	135	135
q7	1113	620	594	594
q8	9415	1987	2043	1987
q9	6684	6214	6175	6175
q10	8396	3531	3510	3510
q11	414	231	239	231
q12	377	213	207	207
q13	17782	2908	2926	2908
q14	271	239	239	239
q15	529	488	475	475
q16	481	383	367	367
q17	955	933	913	913
q18	7272	6506	6371	6371
q19	1592	1548	1543	1543
q20	594	299	331	299
q21	3571	3146	3158	3146
q22	361	300	308	300
Total cold run time: 111265 ms
Total hot run time: 38799 ms
----- Round 2, with runtime_filter_mode=off -----
q1	4087	4061	4019	4019
q2	328	218	218	218
q3	2961	2959	2944	2944
q4	1879	1872	1844	1844
q5	5226	5217	5235	5217
q6	209	128	129	128
q7	2251	1783	1810	1783
q8	3223	3292	3276	3276
q9	8477	8480	8464	8464
q10	3745	3847	3797	3797
q11	558	448	459	448
q12	720	541	569	541
q13	16787	2886	2921	2886
q14	296	248	258	248
q15	508	466	463	463
q16	463	412	406	406
q17	1743	1691	1679	1679
q18	7615	7283	7124	7124
q19	1634	1641	1639	1639
q20	1945	1738	1737	1737
q21	5049	4740	4771	4740
q22	480	410	423	410
Total cold run time: 70184 ms
Total hot run time: 54011 ms

github-actions · 2024-04-07T04:05:37Z

PR approved by anyone and no changes requested.

yiguolei · 2024-04-07T04:13:57Z

run buildall

doris-robot · 2024-04-07T04:32:05Z

TPC-H: Total hot run time: 38662 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 069f45872ea01f507d7b89cdeba9b96d5943d431, data reload: false
------ Round 1 ----------------------------------
q1	17676	4123	4083	4083
q2	2020	191	185	185
q3	10491	1175	1297	1175
q4	10216	859	1045	859
q5	7535	3022	2959	2959
q6	220	137	135	135
q7	1107	628	613	613
q8	9434	1960	2043	1960
q9	8458	6212	6199	6199
q10	8517	3543	3510	3510
q11	423	238	241	238
q12	392	220	218	218
q13	17776	2901	2903	2901
q14	271	248	243	243
q15	538	491	476	476
q16	507	393	371	371
q17	965	917	900	900
q18	7446	6592	6401	6401
q19	1600	1547	1553	1547
q20	570	313	291	291
q21	3504	3104	3103	3103
q22	367	295	306	295
Total cold run time: 110033 ms
Total hot run time: 38662 ms
----- Round 2, with runtime_filter_mode=off -----
q1	4046	4017	4053	4017
q2	325	216	216	216
q3	2978	2975	2965	2965
q4	1886	1880	1846	1846
q5	5272	5228	5246	5228
q6	210	125	129	125
q7	2255	1822	1800	1800
q8	3229	3308	3326	3308
q9	8529	8559	8495	8495
q10	3762	3845	3815	3815
q11	551	451	451	451
q12	697	529	542	529
q13	4653	2927	2909	2909
q14	296	254	263	254
q15	513	472	468	468
q16	438	401	390	390
q17	1707	1669	1675	1669
q18	7879	7285	7153	7153
q19	1654	1644	1649	1644
q20	1949	1720	1738	1720
q21	5065	4809	4755	4755
q22	487	430	429	429
Total cold run time: 58381 ms
Total hot run time: 54186 ms

doris-robot · 2024-04-07T04:43:33Z

TPC-DS: Total hot run time: 180699 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 069f45872ea01f507d7b89cdeba9b96d5943d431, data reload: false
query1	1223	1123	1118	1118
query2	6439	1808	1794	1794
query3	6677	222	227	222
query4	24613	21426	21366	21366
query5	3876	427	411	411
query6	264	199	183	183
query7	4613	302	296	296
query8	234	190	177	177
query9	8470	2200	2214	2200
query10	586	259	247	247
query11	15033	14561	14437	14437
query12	140	89	86	86
query13	1624	383	384	383
query14	8209	6891	6817	6817
query15	209	179	185	179
query16	5637	275	270	270
query17	975	586	579	579
query18	1427	302	281	281
query19	197	156	155	155
query20	98	88	88	88
query21	199	129	129	129
query22	4889	4742	4740	4740
query23	33555	32492	32380	32380
query24	11029	3148	3080	3080
query25	496	387	407	387
query26	728	169	165	165
query27	2960	326	326	326
query28	6732	1821	1814	1814
query29	887	589	596	589
query30	239	171	166	166
query31	946	728	738	728
query32	61	57	55	55
query33	605	259	254	254
query34	901	482	492	482
query35	802	697	701	697
query36	982	854	874	854
query37	111	74	75	74
query38	3517	3358	3356	3356
query39	1558	1527	1536	1527
query40	207	133	130	130
query41	52	49	47	47
query42	120	103	102	102
query43	475	404	414	404
query44	1153	714	717	714
query45	268	250	266	250
query46	1077	820	771	771
query47	1857	1776	1787	1776
query48	380	311	300	300
query49	758	372	362	362
query50	800	396	398	396
query51	6722	6668	6647	6647
query52	101	106	93	93
query53	364	285	289	285
query54	271	234	237	234
query55	88	78	80	78
query56	253	234	233	233
query57	1185	1095	1090	1090
query58	245	215	216	215
query59	2515	2248	2366	2248
query60	261	240	358	240
query61	113	107	106	106
query62	699	443	437	437
query63	311	292	288	288
query64	6335	3261	3299	3261
query65	3089	2990	3045	2990
query66	1462	338	317	317
query67	15415	14939	14956	14939
query68	6581	565	583	565
query69	568	314	304	304
query70	1316	1164	1100	1100
query71	495	280	273	273
query72	6529	2592	2420	2420
query73	838	330	325	325
query74	6699	6457	6356	6356
query75	3598	2332	2264	2264
query76	5702	1214	1244	1214
query77	646	253	251	251
query78	10902	10202	10138	10138
query79	8427	535	535	535
query80	1640	442	422	422
query81	522	248	235	235
query82	736	100	96	96
query83	208	170	171	170
query84	274	88	89	88
query85	1494	282	274	274
query86	480	261	303	261
query87	3651	3473	3465	3465
query88	4248	2368	2362	2362
query89	575	375	380	375
query90	1987	186	191	186
query91	133	103	101	101
query92	61	52	49	49
query93	6369	544	542	542
query94	1140	183	184	183
query95	444	324	316	316
query96	612	276	279	276
query97	2657	2546	2434	2434
query98	228	220	220	220
query99	1221	856	854	854
Total cold run time: 289298 ms
Total hot run time: 180699 ms

doris-robot · 2024-04-07T04:48:58Z

ClickBench: Total hot run time: 29.89 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 069f45872ea01f507d7b89cdeba9b96d5943d431, data reload: false
query1	0.04	0.03	0.03
query2	0.08	0.04	0.04
query3	0.24	0.05	0.05
query4	1.66	0.07	0.07
query5	0.48	0.49	0.50
query6	1.15	0.65	0.66
query7	0.01	0.01	0.02
query8	0.05	0.03	0.04
query9	0.55	0.51	0.50
query10	0.56	0.56	0.56
query11	0.14	0.11	0.12
query12	0.14	0.12	0.12
query13	0.60	0.60	0.59
query14	0.78	0.79	0.81
query15	0.86	0.85	0.83
query16	0.36	0.35	0.36
query17	0.96	1.01	0.98
query18	0.24	0.25	0.26
query19	1.81	1.70	1.70
query20	0.01	0.01	0.01
query21	15.42	0.65	0.65
query22	3.90	7.74	1.47
query23	18.15	1.29	1.24
query24	1.50	0.19	0.18
query25	0.15	0.08	0.08
query26	0.27	0.17	0.17
query27	0.08	0.08	0.08
query28	13.88	0.97	0.96
query29	12.51	3.40	3.38
query30	0.26	0.07	0.06
query31	2.84	0.39	0.40
query32	3.27	0.48	0.48
query33	2.89	2.90	2.84
query34	15.50	4.34	4.34
query35	4.40	4.36	4.35
query36	0.67	0.47	0.49
query37	0.18	0.15	0.15
query38	0.15	0.14	0.14
query39	0.04	0.03	0.03
query40	0.17	0.17	0.15
query41	0.10	0.04	0.04
query42	0.05	0.04	0.05
query43	0.04	0.04	0.04
Total cold run time: 107.14 s
Total hot run time: 29.89 s

doris-robot · 2024-04-07T04:52:04Z

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit 069f45872ea01f507d7b89cdeba9b96d5943d431 with default session variables
Stream load json: 18 seconds loaded 2358488459 Bytes, about 124 MB/s
Stream load orc: 59 seconds loaded 1101869774 Bytes, about 17 MB/s
Stream load parquet: 31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select: 16.3 seconds inserted 10000000 Rows, about 613K ops/s

Some of mysql connector (eg, dotnet MySQL.Data) rely on variable's column type to make connection. eg, `select @@autocommit` should with column type `BIGINT`, not `BIT`, otherwise it will throw error like: ``` System.FormatException: The input string 'True' was not in a correct format. at System.Number.ThrowFormatException[TChar](ReadOnlySpan`1 value) at System.Convert.ToInt32(String value) at MySql.Data.MySqlClient.Driver.LoadCharacterSetsAsync(MySqlConnection connection, Boolean execAsync, CancellationToken cancellationToken) at MySql.Data.MySqlClient.Driver.ConfigureAsync(MySqlConnection connection, Boolean execAsync, CancellationToken cancellationToken) at MySql.Data.MySqlClient.MySqlConnection.OpenAsync(Boolean execAsync, CancellationToken cancellationToken) at MySql.Data.MySqlClient.MySqlConnection.Open() ``` In this PR, I add a new field of `VarAttr`: `convertBoolToLongMethod`, if set, it will convert boolean to long. And set it for `autocommit`

…#33282) Some of mysql connector (eg, dotnet MySQL.Data) rely on variable's column type to make connection. eg, `select @@autocommit` should with column type `BIGINT`, not `BIT`, otherwise it will throw error like: ``` System.FormatException: The input string 'True' was not in a correct format. at System.Number.ThrowFormatException[TChar](ReadOnlySpan`1 value) at System.Convert.ToInt32(String value) at MySql.Data.MySqlClient.Driver.LoadCharacterSetsAsync(MySqlConnection connection, Boolean execAsync, CancellationToken cancellationToken) at MySql.Data.MySqlClient.Driver.ConfigureAsync(MySqlConnection connection, Boolean execAsync, CancellationToken cancellationToken) at MySql.Data.MySqlClient.MySqlConnection.OpenAsync(Boolean execAsync, CancellationToken cancellationToken) at MySql.Data.MySqlClient.MySqlConnection.Open() ``` In this PR, I add a new field of `VarAttr`: `convertBoolToLongMethod`, if set, it will convert boolean to long. And set it for `autocommit`

bp #33887 #33282

) bp apache#33887 apache#33282

…#33282) Some of mysql connector (eg, dotnet MySQL.Data) rely on variable's column type to make connection. eg, `select @@autocommit` should with column type `BIGINT`, not `BIT`, otherwise it will throw error like: ``` System.FormatException: The input string 'True' was not in a correct format. at System.Number.ThrowFormatException[TChar](ReadOnlySpan`1 value) at System.Convert.ToInt32(String value) at MySql.Data.MySqlClient.Driver.LoadCharacterSetsAsync(MySqlConnection connection, Boolean execAsync, CancellationToken cancellationToken) at MySql.Data.MySqlClient.Driver.ConfigureAsync(MySqlConnection connection, Boolean execAsync, CancellationToken cancellationToken) at MySql.Data.MySqlClient.MySqlConnection.OpenAsync(Boolean execAsync, CancellationToken cancellationToken) at MySql.Data.MySqlClient.MySqlConnection.Open() ``` In this PR, I add a new field of `VarAttr`: `convertBoolToLongMethod`, if set, it will convert boolean to long. And set it for `autocommit`

morningman added the kind/behavior-changed label Apr 6, 2024

yiguolei reviewed Apr 7, 2024
View reviewed changes

morningman force-pushed the var_mysql_type branch from 05fd6a6 to 55f262bCompare April 7, 2024 03:42

yiguolei approved these changes Apr 7, 2024
View reviewed changes

github-actionsBot added the approved Indicates a PR has been approved by one committer. label Apr 7, 2024

github-actionsBot added the reviewed label Apr 7, 2024

yiguolei added the usercase Important user case type label label Apr 7, 2024

1
069f458

yiguolei force-pushed the var_mysql_type branch from 55f262b to 069f458Compare April 7, 2024 04:13

wuwenchi approved these changes Apr 7, 2024
View reviewed changes

zddr approved these changes Apr 7, 2024
View reviewed changes

morningman merged commit 119dc4d into apache:masterApr 7, 2024

morningman added the dev/2.0.x label Apr 7, 2024

yiguolei added the dev/2.1.2-merged label Apr 7, 2024

yiguolei mentioned this pull request Apr 8, 2024
Release Note 2.1.2 #33307
Closed

morningman added dev/2.0.x and removed dev/2.0.x labels Apr 10, 2024

This was referenced Apr 19, 2024
[fix](variable) modify @@auto_commit column type to BIGINT in Nereids morningman/doris#2
Closed
[fix](variable) modify @@auto_commit column type to BIGINT in Nereids. #33887
Merged

morningman removed the dev/2.0.x label Apr 19, 2024

zy-kkk added the dev/2.0.x label Jun 4, 2024

morningman added dev/2.0.x and removed dev/2.0.x labels Jun 20, 2024

morningman mentioned this pull request Jun 20, 2024
[fix](variable) modify @@auto_commit column type to BIGINT #36584
Merged

morningman added a commit that referenced this pull request Jun 20, 2024
[fix](variable) modify @@auto_commit column type to BIGINT (#36584)
132b9d6
bp #33887 #33282

xiaokang added dev/2.0.12-merged and removed dev/2.0.x labels Jun 20, 2024

xiaokang mentioned this pull request Jun 23, 2024
Release Note 2.0.12 #36713
Closed

mongo360 pushed a commit to mongo360/doris that referenced this pull request Aug 16, 2024
[fix](variable) modify @@auto_commit column type to BIGINT (apache#36584
e187329
) bp apache#33887 apache#33282

		@@ -3206,14 +3205,14 @@ public void readFromJson(String json) throws IOException {

Uh oh!

Conversation

morningman commented Apr 6, 2024• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Proposed changes

Further comments

Uh oh!

doris-robot commented Apr 6, 2024

Uh oh!

morningman commented Apr 6, 2024

Uh oh!

yiguoleiApr 7, 2024• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

morningmanApr 7, 2024

Choose a reason for hiding this comment

Uh oh!

morningman commented Apr 7, 2024

Uh oh!

yiguolei left a comment

Choose a reason for hiding this comment

Uh oh!

github-actionsBot commented Apr 7, 2024

Uh oh!

doris-robot commented Apr 7, 2024

Uh oh!

github-actionsBot commented Apr 7, 2024

Uh oh!

yiguolei commented Apr 7, 2024

Uh oh!

doris-robot commented Apr 7, 2024

Uh oh!

doris-robot commented Apr 7, 2024

Uh oh!

doris-robot commented Apr 7, 2024

Uh oh!

doris-robot commented Apr 7, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

morningman commented Apr 6, 2024•
edited
Loading

yiguoleiApr 7, 2024•
edited
Loading