[fix](fe) Fix `FrontendHbResponse` serialization compatibility problem by swjtu-zhanglei · Pull Request #30337 · apache/doris

swjtu-zhanglei · 2024-01-24T13:26:25Z

Between branch-1.2-lts and branch-2.1, FrontendHbResponse has upgrade compatiblity problem, because arrowFlightSqlPort field, only metaVerserion less than FeMetaVersion.VERSION_121 will call FrontendHbResponse.readField
Introduced by [feature-wip](arrow-flight)(step2) FE support Arrow Flight server #24314

Proposed changes

Issue Number: close #xxx

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

* Between branch-1.2-lts and branch-2.1, `FrontendHbResponse` has upgrade compatiblity problem, because `arrowFlightSqlPort` field, only metaVerserion less than FeMetaVersion.VERSION_121 will call `FrontendHbResponse.readField` * Introduced by apache#24314

swjtu-zhanglei · 2024-01-24T13:26:43Z

need pick to branch-2.1

swjtu-zhanglei · 2024-01-24T13:26:50Z

run buildall

github-actions · 2024-01-24T13:30:43Z

PR approved by anyone and no changes requested.

doris-robot · 2024-01-24T13:50:06Z

TPC-H: Total hot run time: 38572 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 3bd4e54db46ad3ecab94fb82eef3000e221ffc86, data reload: false
------ Round 1 ----------------------------------
q1	17759	4906	5345	4906
q2	2043	145	139	139
q3	10787	1165	1168	1165
q4	10405	792	800	792
q5	7731	3176	3164	3164
q6	202	124	123	123
q7	883	498	481	481
q8	9227	1901	1939	1901
q9	7182	6395	6339	6339
q10	8199	3093	3085	3085
q11	405	215	209	209
q12	364	189	192	189
q13	18002	3359	3369	3359
q14	250	206	220	206
q15	550	510	508	508
q16	424	367	381	367
q17	929	593	508	508
q18	7671	6980	6795	6795
q19	2148	1386	1342	1342
q20	617	292	313	292
q21	2831	2407	2491	2407
q22	371	319	295	295
Total cold run time: 108980 ms
Total hot run time: 38572 ms
----- Round 2, with runtime_filter_mode=off -----
q1	5651	5531	5401	5401
q2	323	213	205	205
q3	3349	3236	3250	3236
q4	2060	2048	2081	2048
q5	5847	5937	5843	5843
q6	202	121	119	119
q7	2376	1894	1859	1859
q8	3242	3399	3390	3390
q9	9028	8880	8844	8844
q10	3889	3764	3817	3764
q11	558	459	450	450
q12	768	607	658	607
q13	16913	3156	3172	3156
q14	294	257	259	257
q15	573	500	505	500
q16	494	480	470	470
q17	1877	1844	1874	1844
q18	9724	17624	9400	9400
q19	24801	1570	1508	1508
q20	4611	1939	1929	1929
q21	14546	5523	5393	5393
q22	982	536	582	536
Total cold run time: 112108 ms
Total hot run time: 60759 ms

doris-robot · 2024-01-24T14:02:10Z

TPC-DS: Total hot run time: 186322 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 3bd4e54db46ad3ecab94fb82eef3000e221ffc86, data reload: false
query1	940	343	337	337
query2	6545	2025	1823	1823
query3	6705	203	194	194
query4	32786	22402	22224	22224
query5	4452	444	362	362
query6	249	158	158	158
query7	4631	259	255	255
query8	233	175	174	174
query9	8391	2568	2553	2553
query10	410	227	220	220
query11	17219	15779	15476	15476
query12	123	69	65	65
query13	1689	384	371	371
query14	10539	6994	7073	6994
query15	219	177	180	177
query16	5794	252	248	248
query17	959	470	471	470
query18	1798	256	254	254
query19	170	132	132	132
query20	68	71	72	71
query21	189	128	132	128
query22	4905	4800	4620	4620
query23	32310	31390	31077	31077
query24	12587	2828	2756	2756
query25	571	311	303	303
query26	1793	142	141	141
query27	3192	283	279	279
query28	7036	1817	1808	1808
query29	2024	624	607	607
query30	281	139	137	137
query31	950	734	761	734
query32	70	49	49	49
query33	715	214	216	214
query34	1140	454	457	454
query35	859	781	737	737
query36	1330	1185	1136	1136
query37	91	57	62	57
query38	3352	3280	3264	3264
query39	1316	1259	1276	1259
query40	344	88	83	83
query41	34	37	35	35
query42	87	79	84	79
query43	522	439	483	439
query44	1136	683	681	681
query45	192	179	175	175
query46	1073	633	653	633
query47	1703	1602	1560	1560
query48	383	306	316	306
query49	1210	293	285	285
query50	682	310	317	310
query51	5334	5192	5216	5192
query52	92	79	75	75
query53	321	255	256	255
query54	244	192	181	181
query55	81	74	75	74
query56	179	179	173	173
query57	1020	945	941	941
query58	193	164	158	158
query59	2785	2679	2582	2582
query60	206	182	185	182
query61	81	82	81	81
query62	607	369	360	360
query63	279	248	268	248
query64	6115	1807	1750	1750
query65	3332	3242	3252	3242
query66	1402	321	319	319
query67	15649	15008	15115	15008
query68	12546	539	504	504
query69	626	304	294	294
query70	1671	1505	1444	1444
query71	10486	10194	10207	10194
query72	4848	2855	2824	2824
query73	2804	306	307	306
query74	7180	6341	6474	6341
query75	5172	2382	2348	2348
query76	6659	1052	996	996
query77	640	226	240	226
query78	9277	8640	8818	8640
query79	1005	501	505	501
query80	541	330	313	313
query81	448	206	200	200
query82	204	82	81	81
query83	139	115	121	115
query84	279	68	69	68
query85	1091	341	325	325
query86	398	393	368	368
query87	3582	3366	3311	3311
query88	3084	2138	2141	2138
query89	435	383	347	347
query90	1971	189	181	181
query91	162	138	123	123
query92	51	43	48	43
query93	1313	450	417	417
query94	1265	162	157	157
query95	500	454	429	429
query96	626	313	307	307
query97	4255	4174	4161	4161
query98	194	197	191	191
query99	1074	706	704	704
Total cold run time: 308443 ms
Total hot run time: 186322 ms

xinyiZzz

LGTM

github-actions · 2024-01-24T14:05:43Z

PR approved by at least one committer and no changes requested.

doris-robot · 2024-01-24T14:07:36Z

ClickBench: Total hot run time: 30.89 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 3bd4e54db46ad3ecab94fb82eef3000e221ffc86, data reload: false
query1	0.03	0.03	0.02
query2	0.06	0.03	0.02
query3	0.23	0.06	0.06
query4	1.69	0.06	0.06
query5	0.53	0.53	0.53
query6	1.33	0.63	0.63
query7	0.02	0.01	0.01
query8	0.04	0.03	0.02
query9	0.55	0.49	0.50
query10	0.57	0.55	0.54
query11	0.11	0.09	0.09
query12	0.11	0.09	0.08
query13	0.60	0.61	0.61
query14	0.79	0.81	0.80
query15	0.79	0.79	0.77
query16	0.39	0.37	0.38
query17	0.98	1.04	1.05
query18	0.24	0.26	0.23
query19	1.84	1.78	1.77
query20	0.01	0.01	0.01
query21	15.42	0.61	0.57
query22	2.47	2.73	2.29
query23	17.19	0.85	0.68
query24	2.51	0.94	0.46
query25	0.33	0.33	0.14
query26	0.35	0.14	0.13
query27	0.05	0.04	0.06
query28	12.25	0.76	0.77
query29	12.51	3.25	3.25
query30	0.55	0.54	0.53
query31	2.78	0.34	0.34
query32	3.39	0.50	0.48
query33	3.27	3.20	3.24
query34	15.64	4.26	4.33
query35	4.38	4.32	4.29
query36	1.14	1.06	1.06
query37	0.07	0.05	0.05
query38	0.04	0.03	0.03
query39	0.02	0.02	0.01
query40	0.16	0.13	0.13
query41	0.07	0.02	0.01
query42	0.02	0.01	0.01
query43	0.02	0.02	0.02
Total cold run time: 105.54 s
Total hot run time: 30.89 s

doris-robot · 2024-01-24T14:10:41Z

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit 3bd4e54db46ad3ecab94fb82eef3000e221ffc86 with default session variables
Stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s
Stream load orc: 58 seconds loaded 1101869774 Bytes, about 18 MB/s
Stream load parquet: 31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select: 14.7 seconds inserted 10000000 Rows, about 680K ops/s

#30337) * Between branch-1.2-lts and branch-2.1, `FrontendHbResponse` has upgrade compatiblity problem, because `arrowFlightSqlPort` field, only metaVerserion less than FeMetaVersion.VERSION_121 will call `FrontendHbResponse.readField` * Introduced by #24314

* Similar to apache#30337 * Between branch-1.2-lts and branch-2.1, `FrontendHbResponse` has upgrade compatiblity problem, because `arrowFlightSqlPort` field, only metaVerserion less than FeMetaVersion.VERSION_121 will call `BackendHbResponse.readField` * Introduced by apache#24314

…#30441) * Similar to #30337 * Between branch-1.2-lts and branch-2.1, `FrontendHbResponse` has upgrade compatiblity problem, because `arrowFlightSqlPort` field, only metaVerserion less than FeMetaVersion.VERSION_121 will call `BackendHbResponse.readField` * Introduced by #24314

…#30433) * Similar to #30337 * Between branch-1.2-lts and branch-2.1, `FrontendHbResponse` has upgrade compatiblity problem, because `arrowFlightSqlPort` field, only metaVerserion less than FeMetaVersion.VERSION_121 will call `BackendHbResponse.readField` * Introduced by #24314

apache#30337) * Between branch-1.2-lts and branch-2.1, `FrontendHbResponse` has upgrade compatiblity problem, because `arrowFlightSqlPort` field, only metaVerserion less than FeMetaVersion.VERSION_121 will call `FrontendHbResponse.readField` * Introduced by apache#24314

…apache#30433) * Similar to apache#30337 * Between branch-1.2-lts and branch-2.1, `FrontendHbResponse` has upgrade compatiblity problem, because `arrowFlightSqlPort` field, only metaVerserion less than FeMetaVersion.VERSION_121 will call `BackendHbResponse.readField` * Introduced by apache#24314

gavinchou approved these changes Jan 24, 2024
View reviewed changes

github-actionsBot added the reviewed label Jan 24, 2024

xinyiZzz approved these changes Jan 24, 2024
View reviewed changes

github-actionsBot added the approved Indicates a PR has been approved by one committer. label Jan 24, 2024

wm1581066 added the dev/2.1.0 label Jan 24, 2024

yiguolei merged commit aa3ff1e into apache:masterJan 25, 2024

swjtu-zhanglei mentioned this pull request Jan 26, 2024
[fix](fe) Fix BackendHbResponse serialization compatibility problem #30433
Merged

swjtu-zhanglei mentioned this pull request Jan 26, 2024
[fix](fe) Fix BackendHbResponse serialization compatibility problem #30441
Merged

dataroaring added the dev/3.0.0-merged label Aug 5, 2024

Navigation Menu

Uh oh!

[fix](fe) Fix `FrontendHbResponse` serialization compatibility problem - #30337

[fix](fe) Fix `FrontendHbResponse` serialization compatibility problem#30337
yiguolei merged 1 commit into
apache:masterfrom
swjtu-zhanglei:master-fix6

swjtu-zhanglei commented Jan 24, 2024

Uh oh!

swjtu-zhanglei commented Jan 24, 2024

Uh oh!

swjtu-zhanglei commented Jan 24, 2024

Uh oh!

github-actionsBot commented Jan 24, 2024

Uh oh!

doris-robot commented Jan 24, 2024

Uh oh!

doris-robot commented Jan 24, 2024

Uh oh!

xinyiZzz left a comment

Uh oh!

github-actionsBot commented Jan 24, 2024

Uh oh!

doris-robot commented Jan 24, 2024

Uh oh!

doris-robot commented Jan 24, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

Uh oh!

Conversation

swjtu-zhanglei commented Jan 24, 2024

Proposed changes

Further comments

Uh oh!

swjtu-zhanglei commented Jan 24, 2024

Uh oh!

swjtu-zhanglei commented Jan 24, 2024

Uh oh!

github-actionsBot commented Jan 24, 2024

Uh oh!

doris-robot commented Jan 24, 2024

Uh oh!

doris-robot commented Jan 24, 2024

Uh oh!

xinyiZzz left a comment

Choose a reason for hiding this comment

Uh oh!

github-actionsBot commented Jan 24, 2024

Uh oh!

doris-robot commented Jan 24, 2024

Uh oh!

doris-robot commented Jan 24, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants