Skip to content

[chore](ColdHeatSeparation) Reduce the frequency of log following read failure - #32706

Merged
dataroaring merged 2 commits into
apache:masterfrom
ByteYue:log_every_n_follow_read_error
Mar 25, 2024
Merged

[chore](ColdHeatSeparation) Reduce the frequency of log following read failure #32706
dataroaring merged 2 commits into
apache:masterfrom
ByteYue:log_every_n_follow_read_error

Conversation

@ByteYue

@ByteYueByteYue commented Mar 22, 2024

Copy link
Copy Markdown
Contributor

Proposed changes

Issue Number: close #xxx

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

@doris-robot

doris-robot commented Mar 22, 2024

Copy link
Copy Markdown

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@ByteYue

ByteYue commented Mar 22, 2024

Copy link
Copy Markdown
ContributorAuthor

run buildall

github-actions[bot]
github-actionsBot reviewed Mar 22, 2024
View reviewed changes

@github-actionsgithub-actionsBot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

clang-tidy made some suggestions

Comment threadbe/src/olap/olap_server.cpp
@@ -17,6 +17,7 @@

#include <gen_cpp/Types_types.h>

@github-actionsgithub-actionsBotMar 22, 2024

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

warning: 'gen_cpp/Types_types.h' file not found [clang-diagnostic-error]

#include<gen_cpp/Types_types.h>
^

@ByteYue

ByteYue commented Mar 22, 2024

Copy link
Copy Markdown
ContributorAuthor

run buildall

dataroaring
dataroaring approved these changes Mar 22, 2024
View reviewed changes

@dataroaringdataroaring left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions

github-actionsBot commented Mar 22, 2024

Copy link
Copy Markdown
Contributor

PR approved by at least one committer and no changes requested.

@github-actionsgithub-actionsBot added the approved Indicates a PR has been approved by one committer. label Mar 22, 2024
@github-actions

github-actionsBot commented Mar 22, 2024

Copy link
Copy Markdown
Contributor

PR approved by anyone and no changes requested.

@ByteYueByteYue mentioned this pull request Mar 22, 2024
Closed
@doris-robot

doris-robot commented Mar 22, 2024

Copy link
Copy Markdown
TPC-H: Total hot run time: 38343 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 6cd185048c04f6db21633c0daadeefa8cc8b30a8, data reload: false
------ Round 1 ----------------------------------
q1	17747	4494	4290	4290
q2	2480	166	156	156
q3	11257	1151	1253	1151
q4	11123	869	830	830
q5	7692	3125	3086	3086
q6	211	128	127	127
q7	1062	619	588	588
q8	9691	2073	2030	2030
q9	7366	6704	6613	6613
q10	8426	3468	3519	3468
q11	436	228	229	228
q12	370	200	189	189
q13	17802	2818	2854	2818
q14	230	199	209	199
q15	503	463	470	463
q16	523	385	374	374
q17	962	527	602	527
q18	7352	6419	6437	6419
q19	3217	1438	1479	1438
q20	579	260	253	253
q21	3529	2911	2803	2803
q22	355	293	305	293
Total cold run time: 112913 ms
Total hot run time: 38343 ms
----- Round 2, with runtime_filter_mode=off -----
q1	4172	4123	4074	4074
q2	335	233	235	233
q3	2974	2883	2818	2818
q4	1848	1539	1524	1524
q5	5427	5398	5348	5348
q6	194	117	120	117
q7	2241	1865	1870	1865
q8	3184	3306	3299	3299
q9	8753	8693	8764	8693
q10	3825	3780	3765	3765
q11	548	459	444	444
q12	739	536	562	536
q13	16935	2824	2855	2824
q14	283	247	262	247
q15	517	467	456	456
q16	482	417	415	415
q17	1731	1500	1482	1482
q18	7535	7231	7101	7101
q19	1634	1596	1518	1518
q20	1910	1732	1726	1726
q21	4744	4595	4727	4595
q22	540	444	456	444
Total cold run time: 70551 ms
Total hot run time: 53524 ms

@doris-robot

doris-robot commented Mar 22, 2024

Copy link
Copy Markdown
TPC-DS: Total hot run time: 186148 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 6cd185048c04f6db21633c0daadeefa8cc8b30a8, data reload: false
query1	950	367	357	357
query2	7331	2002	1964	1964
query3	6721	217	212	212
query4	31587	21368	21165	21165
query5	4321	513	398	398
query6	270	179	176	176
query7	4656	288	293	288
query8	228	170	171	170
query9	9299	2354	2347	2347
query10	547	254	254	254
query11	17248	14368	14324	14324
query12	142	88	84	84
query13	1617	434	419	419
query14	14134	11109	10872	10872
query15	257	219	193	193
query16	8154	269	270	269
query17	1933	600	555	555
query18	2075	288	286	286
query19	264	163	155	155
query20	92	88	86	86
query21	209	125	130	125
query22	4980	4861	4772	4772
query23	33610	32582	32549	32549
query24	11632	2855	2833	2833
query25	656	389	379	379
query26	1776	160	160	160
query27	3054	351	348	348
query28	7784	1921	1906	1906
query29	1056	653	623	623
query30	309	147	143	143
query31	966	723	736	723
query32	92	57	55	55
query33	777	273	301	273
query34	1054	478	501	478
query35	838	620	607	607
query36	1014	892	872	872
query37	178	79	76	76
query38	3520	3456	3372	3372
query39	1492	1431	1414	1414
query40	298	119	113	113
query41	54	47	48	47
query42	103	96	98	96
query43	488	453	465	453
query44	1150	727	720	720
query45	277	275	269	269
query46	1111	706	687	687
query47	1915	1859	1875	1859
query48	460	355	367	355
query49	1238	354	338	338
query50	769	373	381	373
query51	6606	7241	6665	6665
query52	107	93	96	93
query53	343	274	268	268
query54	334	247	245	245
query55	89	80	79	79
query56	252	227	241	227
query57	1220	1131	1138	1131
query58	240	209	221	209
query59	2688	2582	2674	2582
query60	264	242	244	242
query61	96	92	96	92
query62	653	455	451	451
query63	301	276	270	270
query64	6829	3992	3991	3991
query65	3092	3012	3010	3010
query66	1430	378	352	352
query67	15047	14744	14893	14744
query68	5424	548	523	523
query69	551	376	376	376
query70	1216	1174	1143	1143
query71	414	284	278	278
query72	6208	2729	2525	2525
query73	710	316	326	316
query74	7966	6671	6676	6671
query75	3655	2824	2781	2781
query76	3506	836	954	836
query77	386	264	262	262
query78	10862	10252	10125	10125
query79	7775	526	515	515
query80	1870	402	393	393
query81	550	222	211	211
query82	1147	213	199	199
query83	332	153	150	150
query84	294	80	93	80
query85	1867	373	366	366
query86	487	309	288	288
query87	3669	3488	3509	3488
query88	5314	2385	2372	2372
query89	477	360	364	360
query90	2116	180	179	179
query91	182	148	149	148
query92	68	48	50	48
query93	5729	515	500	500
query94	1335	191	200	191
query95	448	340	331	331
query96	622	266	278	266
query97	3077	2851	2857	2851
query98	249	223	202	202
query99	1204	949	931	931
Total cold run time: 313987 ms
Total hot run time: 186148 ms

@doris-robot

doris-robot commented Mar 22, 2024

Copy link
Copy Markdown

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit 6cd185048c04f6db21633c0daadeefa8cc8b30a8 with default session variables
Stream load json: 18 seconds loaded 2358488459 Bytes, about 124 MB/s
Stream load orc: 59 seconds loaded 1101869774 Bytes, about 17 MB/s
Stream load parquet: 31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select: 21.2 seconds inserted 10000000 Rows, about 471K ops/s

@doris-robot

doris-robot commented Mar 22, 2024

Copy link
Copy Markdown

TeamCity be ut coverage result:
Function Coverage: 35.25% (8731/24766)
Line Coverage: 27.05% (71496/264303)
Region Coverage: 26.28% (37087/141100)
Branch Coverage: 23.19% (18971/81818)
Coverage Report: http://coverage.selectdb-in.cc/coverage/6cd185048c04f6db21633c0daadeefa8cc8b30a8_6cd185048c04f6db21633c0daadeefa8cc8b30a8/report/index.html

@dataroaring
dataroaring merged commit 72e8c62 into apache:masterMar 25, 2024
xiaokang pushed a commit that referenced this pull request Mar 26, 2024
@ByteYue
[chore](ColdHeatSeparation) Reduce the frequency of log following rea…
b1955a9
…d failure #32706 (#32711)
@xiaokangxiaokang mentioned this pull request Mar 26, 2024
Merged
mongo360 pushed a commit to mongo360/doris that referenced this pull request Aug 16, 2024
@ByteYueweixingyu12
[chore](ColdHeatSeparation) Reduce the frequency of log following rea…
508acfc
…d failure apache#32706 (apache#32711)
HappenLee pushed a commit to HappenLee/incubator-doris that referenced this pull request Apr 24, 2026
@ByteYue@HappenLee
[chore](ColdHeatSeparation) Reduce the frequency of log following rea…
495e2e5
…d failure (apache#32706)
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@github-actionsgithub-actions[bot]github-actions[bot] left review comments
@Yukang-LianYukang-LianYukang-Lian approved these changes
@dataroaringdataroaringdataroaring approved these changes

Assignees

No one assigned

Labels

approvedIndicates a PR has been approved by one committer.dev/2.0.8-mergedreviewed

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@ByteYue@doris-robot@Yukang-Lian@dataroaring@xiaokang