Skip to content

branch-3.0: [enhancement](metrics) add some table metrics (#49930) - #50892

Merged
dataroaring merged 1 commit into
apache:branch-3.0from
yagagagaga:auto-pick-49930-branch-3.0
May 14, 2025
Merged

branch-3.0: [enhancement](metrics) add some table metrics (#49930)#50892
dataroaring merged 1 commit into
apache:branch-3.0from
yagagagaga:auto-pick-49930-branch-3.0

Conversation

@yagagagaga

@yagagagagayagagagaga commented May 14, 2025

Copy link
Copy Markdown
Contributor

Cherry-picked from #49930

@yagagagaga
[enhancement](metrics) add some table metrics (apache#49930)
dd24a60
add these metrics in Doris' FE:
```properties
# HELP doris_fe_min_tablet_size_bytes minimal local data size of all (internal) tables' tablets
# TYPE doris_fe_min_tablet_size_bytes gauge
doris_fe_min_tablet_size_bytes 0
# HELP doris_fe_min_partition_size_bytes minimal local data size of all (internal) tables' partitions
# TYPE doris_fe_min_partition_size_bytes gauge
doris_fe_min_partition_size_bytes 0
# HELP doris_fe_min_table_size_bytes minimal local data size of all (internal) tables
# TYPE doris_fe_min_table_size_bytes gauge
doris_fe_min_table_size_bytes 1360
# HELP doris_fe_max_tablet_size_bytes maximal local data size of all (internal) tables' tablets
# TYPE doris_fe_max_tablet_size_bytes gauge
doris_fe_max_tablet_size_bytes 4739
# HELP doris_fe_max_partition_size_bytes maximal local data size of all (internal) tables' partitions
# TYPE doris_fe_max_partition_size_bytes gauge
doris_fe_max_partition_size_bytes 9431
# HELP doris_fe_max_table_size_bytes maximal local data size of all (internal) tables
# TYPE doris_fe_max_table_size_bytes gauge
doris_fe_max_table_size_bytes 9431
# HELP doris_fe_avg_tablet_size_bytes average local data size of all (internal) tables' tablets
# TYPE doris_fe_avg_tablet_size_bytes gauge
doris_fe_avg_tablet_size_bytes 830
# HELP doris_fe_avg_partition_size_bytes average local data size of all (internal) tables' partitions
# TYPE doris_fe_avg_partition_size_bytes gauge
doris_fe_avg_partition_size_bytes 1541
# HELP doris_fe_avg_table_size_bytes average local data size of all (internal) tables
# TYPE doris_fe_avg_table_size_bytes gauge
doris_fe_avg_table_size_bytes 5395
```
@hello-stephen

hello-stephen commented May 14, 2025

Copy link
Copy Markdown
Contributor

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@yagagagagayagagagaga reopened this May 14, 2025
@yagagagaga

yagagagaga commented May 14, 2025

Copy link
Copy Markdown
ContributorAuthor

run buildall

@doris-robot

doris-robot commented May 14, 2025

Copy link
Copy Markdown
TPC-H: Total hot run time: 39816 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit dd24a6075848302c85e5c61a44da202eff9c6e26, data reload: false
------ Round 1 ----------------------------------
q1	17570	7033	6578	6578
q2	2067	172	164	164
q3	10602	1116	1139	1116
q4	10560	757	720	720
q5	7771	2838	2843	2838
q6	219	135	131	131
q7	960	623	613	613
q8	9338	1963	2012	1963
q9	6568	6364	6377	6364
q10	7035	2249	2279	2249
q11	466	259	249	249
q12	410	209	201	201
q13	17769	2953	2994	2953
q14	230	202	204	202
q15	498	456	463	456
q16	658	587	582	582
q17	963	546	564	546
q18	7351	6570	6768	6570
q19	1388	1009	1041	1009
q20	475	198	192	192
q21	4010	3147	3318	3147
q22	1091	994	973	973
Total cold run time: 107999 ms
Total hot run time: 39816 ms
----- Round 2, with runtime_filter_mode=off -----
q1	6664	6622	6609	6609
q2	332	230	233	230
q3	2884	2786	2894	2786
q4	2086	1824	1795	1795
q5	5851	5789	5765	5765
q6	212	134	125	125
q7	2219	1841	1814	1814
q8	3397	3657	3588	3588
q9	9038	8898	9134	8898
q10	3609	3559	3589	3559
q11	611	551	529	529
q12	842	655	648	648
q13	9422	3324	3329	3324
q14	315	310	312	310
q15	524	477	461	461
q16	706	653	659	653
q17	1860	1621	1605	1605
q18	8403	7820	7878	7820
q19	1750	1560	1524	1524
q20	2123	1890	1877	1877
q21	5571	5112	5284	5112
q22	1104	1059	1035	1035
Total cold run time: 69523 ms
Total hot run time: 60067 ms

@doris-robot

doris-robot commented May 14, 2025

Copy link
Copy Markdown
TPC-DS: Total hot run time: 196942 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit dd24a6075848302c85e5c61a44da202eff9c6e26, data reload: false
query1	1273	916	891	891
query2	6396	2150	2083	2083
query3	10834	4357	4283	4283
query4	61242	28971	23496	23496
query5	5269	464	445	445
query6	400	175	183	175
query7	5450	321	307	307
query8	324	203	217	203
query9	8486	2598	2581	2581
query10	439	264	253	253
query11	17387	15287	16279	15287
query12	169	111	110	110
query13	1414	462	408	408
query14	10653	6790	7265	6790
query15	202	185	180	180
query16	7141	493	508	493
query17	1191	589	579	579
query18	1925	344	327	327
query19	214	161	160	160
query20	119	117	111	111
query21	207	101	104	101
query22	4722	4438	4657	4438
query23	34610	33909	34227	33909
query24	6133	2891	2974	2891
query25	514	384	399	384
query26	647	169	173	169
query27	1790	352	369	352
query28	4303	2443	2443	2443
query29	733	435	441	435
query30	243	165	173	165
query31	1007	824	848	824
query32	66	56	55	55
query33	438	281	295	281
query34	924	506	525	506
query35	860	753	742	742
query36	1084	961	976	961
query37	119	82	71	71
query38	4020	3945	3976	3945
query39	1557	1456	1466	1456
query40	201	97	94	94
query41	49	46	48	46
query42	116	96	99	96
query43	538	481	496	481
query44	1174	841	837	837
query45	181	180	171	171
query46	1156	738	719	719
query47	2032	1897	1917	1897
query48	498	382	402	382
query49	724	412	412	412
query50	851	423	415	415
query51	7312	7213	7255	7213
query52	103	90	90	90
query53	269	181	186	181
query54	577	472	452	452
query55	79	78	79	78
query56	258	258	252	252
query57	1259	1148	1170	1148
query58	215	211	213	211
query59	3467	3284	3185	3185
query60	278	261	287	261
query61	149	108	126	108
query62	799	689	686	686
query63	217	198	194	194
query64	1412	678	646	646
query65	3289	3199	3196	3196
query66	727	318	310	310
query67	16089	15703	15401	15401
query68	4161	582	585	582
query69	432	282	269	269
query70	1231	1092	1068	1068
query71	343	270	261	261
query72	6520	4185	4138	4138
query73	755	355	356	355
query74	10255	9004	9163	9004
query75	3395	2620	2672	2620
query76	2035	1004	1126	1004
query77	488	265	273	265
query78	10639	9707	9528	9528
query79	2292	588	609	588
query80	1359	415	426	415
query81	536	244	239	239
query82	1002	91	86	86
query83	161	139	145	139
query84	285	82	75	75
query85	988	307	292	292
query86	394	300	301	300
query87	4392	4217	4274	4217
query88	4188	2395	2346	2346
query89	406	298	290	290
query90	1971	186	182	182
query91	186	147	144	144
query92	63	49	50	49
query93	2649	564	558	558
query94	806	294	285	285
query95	351	250	244	244
query96	639	281	275	275
query97	3282	3183	3156	3156
query98	213	206	198	198
query99	1653	1334	1318	1318
Total cold run time: 316594 ms
Total hot run time: 196942 ms

@doris-robot

doris-robot commented May 14, 2025

Copy link
Copy Markdown
ClickBench: Total hot run time: 32.04 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit dd24a6075848302c85e5c61a44da202eff9c6e26, data reload: false
query1	0.03	0.04	0.03
query2	0.08	0.02	0.03
query3	0.24	0.06	0.06
query4	1.62	0.11	0.10
query5	0.50	0.49	0.49
query6	1.14	0.73	0.72
query7	0.02	0.01	0.01
query8	0.04	0.03	0.04
query9	0.58	0.50	0.50
query10	0.56	0.55	0.57
query11	0.14	0.10	0.11
query12	0.17	0.11	0.11
query13	0.61	0.60	0.59
query14	2.78	2.76	2.73
query15	0.90	0.82	0.83
query16	0.40	0.37	0.37
query17	0.99	1.07	0.97
query18	0.23	0.21	0.21
query19	1.93	1.82	2.01
query20	0.02	0.01	0.01
query21	15.38	0.59	0.59
query22	2.50	2.04	2.20
query23	17.01	0.95	0.76
query24	3.05	0.79	1.54
query25	0.18	0.11	0.28
query26	0.44	0.14	0.12
query27	0.05	0.05	0.04
query28	10.21	0.50	0.47
query29	12.58	3.26	3.25
query30	0.25	0.06	0.06
query31	2.87	0.39	0.39
query32	3.24	0.46	0.45
query33	2.96	3.01	3.01
query34	17.08	4.49	4.48
query35	4.61	4.48	4.52
query36	0.66	0.48	0.46
query37	0.09	0.06	0.07
query38	0.04	0.03	0.04
query39	0.04	0.03	0.02
query40	0.16	0.12	0.13
query41	0.08	0.02	0.02
query42	0.04	0.02	0.03
query43	0.04	0.03	0.03
Total cold run time: 106.54 s
Total hot run time: 32.04 s

dataroaring
dataroaring approved these changes May 14, 2025
View reviewed changes

@dataroaringdataroaring left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@dataroaring
dataroaring merged commit 610a37c into apache:branch-3.0May 14, 2025
@yagagagaga
yagagagaga deleted the auto-pick-49930-branch-3.0 branch May 15, 2025 04:07
Hastyshell pushed a commit to Hastyshell/doris that referenced this pull request Jul 21, 2025
@yagagagaga
branch-3.0: [enhancement](metrics) add some table metrics (apache#49930…
84e0ec9
…) (apache#4054)
Cherry-picked from apache#50892
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@dataroaringdataroaringdataroaring approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@yagagagaga@hello-stephen@doris-robot@dataroaring