[chore](routine-load) make get Kafka meta timeout configurable by sollhui · Pull Request #36619 · apache/doris

sollhui · 2024-06-20T10:59:43Z

Sometimes, the delay from be to Kafka is relatively high, which can cause get info RPC timeout.

Make get Kafka meta timeout configurable allow users to customize timeout times which can solve this issue.

doris-robot · 2024-06-20T10:59:48Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

sollhui · 2024-06-20T11:00:51Z

run buildall

github-actions · 2024-06-20T11:05:43Z

clang-tidy review says "All clean, LGTM! 👍"

github-actions · 2024-06-20T11:06:14Z

clang-tidy review says "All clean, LGTM! 👍"

doris-robot · 2024-06-20T11:18:11Z

TeamCity be ut coverage result:
Function Coverage: 36.48% (9009/24695)
Line Coverage: 28.03% (73893/263629)
Region Coverage: 27.51% (38373/139486)
Branch Coverage: 24.21% (19563/80790)
Coverage Report: http://coverage.selectdb-in.cc/coverage/93008f67eed2925a56219ad8490c3391b908a650_93008f67eed2925a56219ad8490c3391b908a650/report/index.html

doris-robot · 2024-06-20T11:29:41Z

TPC-H: Total hot run time: 39856 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 93008f67eed2925a56219ad8490c3391b908a650, data reload: false
------ Round 1 ----------------------------------
q1	17611	4384	4382	4382
q2	2010	188	197	188
q3	10469	1082	1078	1078
q4	10180	807	764	764
q5	7496	2638	2628	2628
q6	222	137	150	137
q7	963	605	596	596
q8	9223	2058	2035	2035
q9	8947	6476	6432	6432
q10	8996	3743	3754	3743
q11	464	243	245	243
q12	498	232	228	228
q13	17769	3008	2983	2983
q14	275	231	217	217
q15	524	486	489	486
q16	525	377	381	377
q17	978	658	707	658
q18	7938	7378	7346	7346
q19	5575	1560	1526	1526
q20	654	309	322	309
q21	5001	3163	3164	3163
q22	384	337	349	337
Total cold run time: 116702 ms
Total hot run time: 39856 ms
----- Round 2, with runtime_filter_mode=off -----
q1	4374	4241	4350	4241
q2	386	271	275	271
q3	2991	2807	2880	2807
q4	1991	1729	1691	1691
q5	5635	5466	5452	5452
q6	225	131	131	131
q7	2253	1924	1866	1866
q8	3274	3412	3448	3412
q9	8736	8710	8745	8710
q10	4088	3853	3730	3730
q11	602	511	503	503
q12	813	634	629	629
q13	16985	3138	3194	3138
q14	304	271	271	271
q15	531	476	497	476
q16	496	442	438	438
q17	1800	1559	1489	1489
q18	8158	7827	7795	7795
q19	1801	1625	1632	1625
q20	2441	1925	1844	1844
q21	5112	4923	4919	4919
q22	618	543	546	543
Total cold run time: 73614 ms
Total hot run time: 55981 ms

doris-robot · 2024-06-20T11:40:53Z

TPC-DS: Total hot run time: 174117 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 93008f67eed2925a56219ad8490c3391b908a650, data reload: false
query1	945	386	378	378
query2	6456	2426	2336	2336
query3	6642	209	212	209
query4	19021	17622	17505	17505
query5	3631	501	470	470
query6	257	150	162	150
query7	4598	293	289	289
query8	312	298	297	297
query9	8688	2416	2398	2398
query10	559	301	285	285
query11	10377	10148	10023	10023
query12	121	85	83	83
query13	1659	378	380	378
query14	9413	6895	7696	6895
query15	236	193	201	193
query16	7750	278	268	268
query17	1869	555	535	535
query18	1899	278	298	278
query19	204	158	160	158
query20	93	83	80	80
query21	203	132	127	127
query22	4551	4533	4554	4533
query23	34206	33543	33682	33543
query24	11081	2811	2772	2772
query25	601	346	358	346
query26	701	147	148	147
query27	2227	322	315	315
query28	6006	2077	2038	2038
query29	845	604	608	604
query30	249	156	148	148
query31	948	739	744	739
query32	88	52	51	51
query33	745	274	272	272
query34	886	478	456	456
query35	741	596	608	596
query36	1081	916	952	916
query37	135	67	68	67
query38	2876	2753	2745	2745
query39	841	779	805	779
query40	212	125	118	118
query41	59	51	50	50
query42	114	96	99	96
query43	595	558	532	532
query44	1210	723	726	723
query45	196	156	161	156
query46	1068	704	721	704
query47	1871	1836	1799	1799
query48	362	304	296	296
query49	847	401	448	401
query50	755	398	378	378
query51	7021	6679	6648	6648
query52	105	96	94	94
query53	356	290	286	286
query54	883	455	451	451
query55	75	73	75	73
query56	301	268	273	268
query57	1133	1068	1066	1066
query58	266	251	258	251
query59	3228	3137	3005	3005
query60	313	289	283	283
query61	115	112	110	110
query62	626	465	448	448
query63	318	287	288	287
query64	8625	2369	1835	1835
query65	3173	3123	3084	3084
query66	761	329	349	329
query67	15513	15337	14850	14850
query68	8535	554	546	546
query69	670	490	427	427
query70	1374	1083	1193	1083
query71	509	280	278	278
query72	7652	5857	5867	5857
query73	2137	320	322	320
query74	5901	5522	5502	5502
query75	4182	2710	2724	2710
query76	4344	955	920	920
query77	647	307	313	307
query78	10317	9782	9652	9652
query79	8212	511	518	511
query80	1400	459	449	449
query81	539	221	216	216
query82	264	102	107	102
query83	199	166	165	165
query84	270	88	84	84
query85	860	286	267	267
query86	340	300	329	300
query87	3272	3093	3061	3061
query88	4538	2340	2327	2327
query89	508	381	369	369
query90	1938	183	183	183
query91	159	98	100	98
query92	60	49	50	49
query93	5850	498	497	497
query94	1264	188	185	185
query95	395	312	310	310
query96	604	267	259	259
query97	3196	3086	3032	3032
query98	208	197	196	196
query99	1174	832	852	832
Total cold run time: 285360 ms
Total hot run time: 174117 ms

doris-robot · 2024-06-20T11:46:24Z

ClickBench: Total hot run time: 30.94 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 93008f67eed2925a56219ad8490c3391b908a650, data reload: false
query1	0.04	0.03	0.03
query2	0.09	0.04	0.04
query3	0.23	0.06	0.05
query4	1.67	0.08	0.07
query5	0.50	0.49	0.53
query6	1.13	0.72	0.72
query7	0.02	0.01	0.02
query8	0.06	0.05	0.05
query9	0.54	0.48	0.50
query10	0.54	0.53	0.53
query11	0.16	0.12	0.11
query12	0.15	0.12	0.12
query13	0.60	0.58	0.61
query14	0.77	0.77	0.76
query15	0.82	0.81	0.84
query16	0.36	0.37	0.37
query17	0.96	1.03	1.04
query18	0.22	0.25	0.23
query19	1.79	1.83	1.70
query20	0.01	0.01	0.02
query21	15.41	0.66	0.65
query22	3.85	7.11	2.32
query23	18.29	1.41	1.25
query24	2.08	0.22	0.24
query25	0.15	0.09	0.08
query26	0.27	0.18	0.17
query27	0.08	0.08	0.08
query28	13.26	1.02	0.99
query29	12.57	3.31	3.27
query30	0.26	0.07	0.06
query31	2.86	0.40	0.38
query32	3.28	0.48	0.46
query33	2.84	2.90	2.94
query34	17.14	4.43	4.41
query35	4.45	4.50	4.44
query36	0.65	0.46	0.47
query37	0.19	0.16	0.17
query38	0.16	0.15	0.15
query39	0.05	0.03	0.04
query40	0.17	0.15	0.14
query41	0.09	0.05	0.05
query42	0.06	0.04	0.05
query43	0.04	0.04	0.04
Total cold run time: 108.86 s
Total hot run time: 30.94 s

liaoxin01

LGTM

github-actions · 2024-06-21T15:08:27Z

PR approved by at least one committer and no changes requested.

github-actions · 2024-06-21T15:08:30Z

PR approved by anyone and no changes requested.

dataroaring

LGTM

Sometimes, the delay from be to Kafka is relatively high, which can cause get info RPC timeout. Make get Kafka meta timeout configurable allow users to customize timeout times which can solve this issue.

…37399) pick #36619

…e#36619) Sometimes, the delay from be to Kafka is relatively high, which can cause get info RPC timeout. Make get Kafka meta timeout configurable allow users to customize timeout times which can solve this issue.

make get Kafka meta timeout configurable
93008f6

liaoxin01 approved these changes Jun 21, 2024
View reviewed changes

liaoxin01 added the dev/2.1.x label Jun 21, 2024

github-actionsBot added the approved Indicates a PR has been approved by one committer. label Jun 21, 2024

github-actionsBot added the reviewed label Jun 21, 2024

liaoxin01 added the dev/2.0.x label Jun 21, 2024

dataroaring approved these changes Jun 21, 2024
View reviewed changes

dataroaring merged commit 179e650 into apache:masterJun 22, 2024

sollhui mentioned this pull request Jul 7, 2024
[branch-2.1](routine-load) make get Kafka meta timeout configurable #37399
Merged

dataroaring pushed a commit that referenced this pull request Jul 8, 2024
[branch-2.1](routine-load) make get Kafka meta timeout configurable (#…
dd18652
…37399) pick #36619

dataroaring added dev/2.1.5-merged and removed dev/2.1.x labels Jul 8, 2024

sollhui mentioned this pull request Jul 8, 2024
[branch-2.0](routine-load) add retry when get Kafka meta info and make get Kafka meta timeout configurable #37458
Merged

liaoxin01 added dev/2.0.13-merged and removed dev/2.0.x labels Jul 8, 2024

dataroaring added the dev/3.0.0-merged label Aug 5, 2024

Navigation Menu

Uh oh!

[chore](routine-load) make get Kafka meta timeout configurable - #36619

[chore](routine-load) make get Kafka meta timeout configurable#36619
dataroaring merged 1 commit into
apache:masterfrom
sollhui:routine_load_get_meta_timeout

sollhui commented Jun 20, 2024•
edited
Loading

Uh oh!

doris-robot commented Jun 20, 2024

Uh oh!

sollhui commented Jun 20, 2024

Uh oh!

github-actionsBot commented Jun 20, 2024

Uh oh!

github-actionsBot commented Jun 20, 2024

Uh oh!

doris-robot commented Jun 20, 2024

Uh oh!

doris-robot commented Jun 20, 2024

Uh oh!

doris-robot commented Jun 20, 2024

Uh oh!

doris-robot commented Jun 20, 2024

Uh oh!

liaoxin01 left a comment

Uh oh!

github-actionsBot commented Jun 21, 2024

Uh oh!

github-actionsBot commented Jun 21, 2024

Uh oh!

dataroaring left a comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Uh oh!

Conversation

sollhui commented Jun 20, 2024• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

doris-robot commented Jun 20, 2024

Uh oh!

sollhui commented Jun 20, 2024

Uh oh!

github-actionsBot commented Jun 20, 2024

Uh oh!

github-actionsBot commented Jun 20, 2024

Uh oh!

doris-robot commented Jun 20, 2024

Uh oh!

doris-robot commented Jun 20, 2024

Uh oh!

doris-robot commented Jun 20, 2024

Uh oh!

doris-robot commented Jun 20, 2024

Uh oh!

liaoxin01 left a comment

Choose a reason for hiding this comment

Uh oh!

github-actionsBot commented Jun 21, 2024

Uh oh!

github-actionsBot commented Jun 21, 2024

Uh oh!

dataroaring left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

sollhui commented Jun 20, 2024•
edited
Loading