Skip to content

[fix](group commit) Fix group commit debug log and improve performance - #38754

Merged
dataroaring merged 2 commits into
apache:masterfrom
mymeiyi:0801
Aug 2, 2024
Merged

[fix](group commit) Fix group commit debug log and improve performance#38754
dataroaring merged 2 commits into
apache:masterfrom
mymeiyi:0801

Conversation

@mymeiyi

@mymeiyimymeiyi commented Aug 2, 2024

Copy link
Copy Markdown
Contributor

Proposed changes

  1. show query_id in the debug log of when group commit insert does not work
  2. remove string.format to improve performance

@doris-robot

doris-robot commented Aug 2, 2024

Copy link
Copy Markdown

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@mymeiyi

mymeiyi commented Aug 2, 2024

Copy link
Copy Markdown
ContributorAuthor

run buildall

@doris-robot

doris-robot commented Aug 2, 2024

Copy link
Copy Markdown
TPC-H: Total hot run time: 41835 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit ec25008eb0dff8397e7b8a60e155f6d41e505ea5, data reload: false
------ Round 1 ----------------------------------
q1	17610	4169	4086	4086
q2	2023	199	194	194
q3	10901	1313	1392	1313
q4	10748	888	940	888
q5	8187	3048	3032	3032
q6	230	143	139	139
q7	1091	644	619	619
q8	10353	1909	1995	1909
q9	8662	6818	6603	6603
q10	8753	3857	3856	3856
q11	428	248	255	248
q12	406	227	219	219
q13	17763	2911	2935	2911
q14	273	248	240	240
q15	523	475	486	475
q16	521	388	386	386
q17	980	920	934	920
q18	8207	7421	7295	7295
q19	1441	1234	1220	1220
q20	553	319	339	319
q21	5297	4684	4741	4684
q22	359	287	279	279
Total cold run time: 115309 ms
Total hot run time: 41835 ms
----- Round 2, with runtime_filter_mode=off -----
q1	4091	4041	4045	4041
q2	326	216	217	216
q3	2991	3138	3207	3138
q4	2038	2042	1992	1992
q5	5485	5463	5390	5390
q6	219	128	128	128
q7	2191	1790	1812	1790
q8	3287	3384	3373	3373
q9	8819	8744	8436	8436
q10	3959	4025	4029	4025
q11	558	452	469	452
q12	778	561	586	561
q13	12004	3123	3140	3123
q14	304	277	278	277
q15	558	481	493	481
q16	466	411	411	411
q17	1798	1749	1755	1749
q18	8354	7742	7638	7638
q19	1751	1731	1733	1731
q20	2030	1837	1828	1828
q21	5563	5386	5537	5386
q22	522	459	475	459
Total cold run time: 68092 ms
Total hot run time: 56625 ms

@doris-robot

doris-robot commented Aug 2, 2024

Copy link
Copy Markdown
TPC-DS: Total hot run time: 169228 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit ec25008eb0dff8397e7b8a60e155f6d41e505ea5, data reload: false
query1	909	379	372	372
query2	6306	1645	1625	1625
query3	6679	211	214	211
query4	19677	17458	17127	17127
query5	3705	515	511	511
query6	271	168	168	168
query7	4604	291	286	286
query8	250	190	191	190
query9	8506	2386	2352	2352
query10	431	283	273	273
query11	10520	10051	10078	10051
query12	119	89	87	87
query13	1632	381	378	378
query14	9972	6818	7344	6818
query15	201	164	163	163
query16	7030	476	417	417
query17	941	569	548	548
query18	1911	283	276	276
query19	187	143	138	138
query20	112	83	83	83
query21	203	102	98	98
query22	4193	3989	3970	3970
query23	33731	33751	33638	33638
query24	10462	3061	3087	3061
query25	692	392	401	392
query26	1699	154	153	153
query27	3118	290	280	280
query28	7546	1990	2004	1990
query29	1309	428	461	428
query30	246	160	154	154
query31	945	756	774	756
query32	102	59	58	58
query33	687	319	329	319
query34	928	509	516	509
query35	878	784	756	756
query36	1031	893	876	876
query37	298	80	75	75
query38	2885	2789	2747	2747
query39	869	813	838	813
query40	253	117	109	109
query41	48	44	44	44
query42	142	105	107	105
query43	457	412	419	412
query44	1197	735	745	735
query45	208	176	173	173
query46	1080	840	798	798
query47	1842	1733	1726	1726
query48	371	298	296	296
query49	932	418	427	418
query50	893	451	447	447
query51	6723	6727	6609	6609
query52	101	91	89	89
query53	252	179	179	179
query54	634	457	463	457
query55	75	77	74	74
query56	269	270	255	255
query57	1151	1035	1029	1029
query58	270	277	281	277
query59	2592	2405	2470	2405
query60	305	284	279	279
query61	99	95	94	94
query62	883	657	671	657
query63	223	181	177	177
query64	5653	1897	1898	1897
query65	3169	3086	3090	3086
query66	1311	335	328	328
query67	15418	14738	14874	14738
query68	4241	562	578	562
query69	440	301	302	301
query70	1125	1079	1066	1066
query71	432	275	293	275
query72	7196	2682	2484	2484
query73	763	334	331	331
query74	6058	5634	5670	5634
query75	3397	2791	2713	2713
query76	2256	1211	1249	1211
query77	425	302	327	302
query78	9391	9003	8892	8892
query79	1476	536	524	524
query80	1143	511	494	494
query81	583	226	232	226
query82	1066	131	127	127
query83	248	165	164	164
query84	268	77	80	77
query85	1204	320	314	314
query86	401	302	311	302
query87	3250	3100	3079	3079
query88	3028	2542	2501	2501
query89	382	297	294	294
query90	1689	191	190	190
query91	187	100	100	100
query92	63	51	50	50
query93	1402	616	607	607
query94	771	306	301	301
query95	380	274	268	268
query96	606	282	285	282
query97	3232	3044	3068	3044
query98	211	201	196	196
query99	1617	1277	1304	1277
Total cold run time: 261267 ms
Total hot run time: 169228 ms

@doris-robot

doris-robot commented Aug 2, 2024

Copy link
Copy Markdown
ClickBench: Total hot run time: 30.19 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit ec25008eb0dff8397e7b8a60e155f6d41e505ea5, data reload: false
query1	0.05	0.04	0.04
query2	0.07	0.04	0.04
query3	0.23	0.05	0.05
query4	1.68	0.06	0.07
query5	0.49	0.48	0.49
query6	1.13	0.72	0.72
query7	0.02	0.02	0.01
query8	0.05	0.04	0.04
query9	0.58	0.51	0.50
query10	0.57	0.57	0.56
query11	0.15	0.12	0.11
query12	0.15	0.12	0.12
query13	0.60	0.60	0.60
query14	0.77	0.80	0.79
query15	0.93	0.86	0.84
query16	0.34	0.34	0.34
query17	1.00	1.02	1.02
query18	0.21	0.21	0.20
query19	1.82	1.76	1.69
query20	0.02	0.01	0.01
query21	15.39	0.78	0.66
query22	4.16	7.57	1.60
query23	18.11	1.28	1.33
query24	2.25	0.21	0.21
query25	0.18	0.08	0.08
query26	0.31	0.21	0.22
query27	0.46	0.23	0.23
query28	13.17	0.99	0.97
query29	12.53	3.30	3.30
query30	0.26	0.06	0.07
query31	2.85	0.41	0.40
query32	3.27	0.48	0.48
query33	2.92	2.98	2.95
query34	15.45	4.25	4.21
query35	4.30	4.26	4.30
query36	0.68	0.48	0.48
query37	0.19	0.16	0.15
query38	0.16	0.15	0.15
query39	0.04	0.03	0.04
query40	0.16	0.13	0.13
query41	0.10	0.05	0.05
query42	0.05	0.04	0.05
query43	0.05	0.04	0.04
Total cold run time: 107.9 s
Total hot run time: 30.19 s

dataroaring
dataroaring approved these changes Aug 2, 2024
View reviewed changes

@dataroaringdataroaring left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions

github-actionsBot commented Aug 2, 2024

Copy link
Copy Markdown
Contributor

PR approved by at least one committer and no changes requested.

@github-actionsgithub-actionsBot added approved Indicates a PR has been approved by one committer. reviewed labels Aug 2, 2024
@github-actions

github-actionsBot commented Aug 2, 2024

Copy link
Copy Markdown
Contributor

PR approved by anyone and no changes requested.

@dataroaring
dataroaring merged commit f18d7b6 into apache:masterAug 2, 2024
mymeiyi added a commit to mymeiyi/doris that referenced this pull request Aug 5, 2024
@mymeiyi
[fix](group commit) Fix group commit debug log and improve performance (
acc56a3
apache#38754)
1. show `query_id` in the debug log of when group commit insert does not
work
2. remove string.format to improve performance
@mymeiyimymeiyi mentioned this pull request Aug 5, 2024
Merged
yiguolei pushed a commit to mymeiyi/doris that referenced this pull request Aug 5, 2024
@mymeiyi@yiguolei
[fix](group commit) Fix group commit debug log and improve performance (
3b2fe18
apache#38754)
1. show `query_id` in the debug log of when group commit insert does not
work
2. remove string.format to improve performance
yiguolei pushed a commit that referenced this pull request Aug 5, 2024
@mymeiyi
[fix](group commit) Fix group commit debug log and improve performance (
ce75e6a
#38754) (#38841)
Pick #38754
dataroaring pushed a commit that referenced this pull request Aug 11, 2024
@mymeiyi@dataroaring
[fix](group commit) Fix group commit debug log and improve performance (
3c39d64
#38754)
## Proposed changes
1. show `query_id` in the debug log of when group commit insert does not
work
2. remove string.format to improve performance
dataroaring pushed a commit that referenced this pull request Aug 16, 2024
@mymeiyi@dataroaring
[fix](group commit) Fix group commit debug log and improve performance (
e2087f2
#38754)
## Proposed changes
1. show `query_id` in the debug log of when group commit insert does not
work
2. remove string.format to improve performance
HappenLee pushed a commit to HappenLee/incubator-doris that referenced this pull request Apr 24, 2026
@mymeiyi@HappenLee
[fix](group commit) Fix group commit debug log and improve performance (
d88d97a
apache#38754)
## Proposed changes
1. show `query_id` in the debug log of when group commit insert does not
work
2. remove string.format to improve performance
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@Yukang-LianYukang-LianYukang-Lian approved these changes
@dataroaringdataroaringdataroaring approved these changes

Assignees

No one assigned

Labels

approvedIndicates a PR has been approved by one committer.dev/2.1.6-mergeddev/3.0.2-mergedreviewed

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@mymeiyi@doris-robot@Yukang-Lian@dataroaring@yiguolei