Skip to content

[Improment]Add workload group resource usage - #39177

Merged
yiguolei merged 2 commits into
apache:masterfrom
wangbo:0809_wg_monitor
Aug 12, 2024
Merged

[Improment]Add workload group resource usage#39177
yiguolei merged 2 commits into
apache:masterfrom
wangbo:0809_wg_monitor

Conversation

@wangbo

@wangbowangbo commented Aug 9, 2024

Copy link
Copy Markdown
Contributor

Proposed changes

mysql [information_schema]>select BE_ID,WORKLOAD_GROUP_ID,CPU_USAGE,LOCAL_SCAN_BYTES_PER_SECOND/1024/1024 as scan_io_mb, MEMORY_USAGE_BYTES/1024/1024 mem_mb from workload_group_resource_usage;
+-------+-------------------+-----------+--------------------+-------------------+
| BE_ID | WORKLOAD_GROUP_ID | CPU_USAGE | scan_io_mb | mem_mb |
+-------+-------------------+-----------+--------------------+-------------------+
| 10005 | 62053 | 61.41% | 1516.4589414596558 | 27970.84313774109 |
| 10005 | 1 | 0.00% | 0 | 0 |
+-------+-------------------+-----------+--------------------+-------------------+

@doris-robot

doris-robot commented Aug 9, 2024

Copy link
Copy Markdown

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

github-actions[bot]
github-actionsBot reviewed Aug 9, 2024
View reviewed changes

@github-actionsgithub-actionsBot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

clang-tidy made some suggestions

Comment threadbe/src/exec/schema_scanner/schema_workload_group_resource_usage_scanner.cpp
Comment threadbe/src/exec/schema_scanner/schema_workload_group_resource_usage_scanner.cpp
Comment threadbe/src/runtime/workload_group/workload_group.h
Comment threadbe/src/runtime/workload_group/workload_group_manager.cpp
Comment threadbe/src/runtime/workload_management/io_throttle.cpp
Comment threadbe/src/runtime/workload_management/io_throttle.h
yiguolei
yiguolei reviewed Aug 10, 2024
View reviewed changes
Comment threadbe/src/exec/schema_scanner/schema_workload_group_resource_usage_scanner.cpp Outdated
{"BE_ID", TYPE_BIGINT, sizeof(int64_t), false},
{"WORKLOAD_GROUP_ID", TYPE_BIGINT, sizeof(int64_t), false},
{"MEMORY_USAGE_BYTES", TYPE_BIGINT, sizeof(int64_t), false},
{"CPU_USAGE", TYPE_VARCHAR, sizeof(StringRef), false},

@yiguoleiyiguoleiAug 10, 2024

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

这个可能不行，不行就把百分号去掉。否则用户没法排序

yiguolei
yiguolei reviewed Aug 10, 2024
View reviewed changes
Comment threadbe/src/runtime/workload_management/io_throttle.cpp
yiguolei
yiguolei reviewed Aug 10, 2024
View reviewed changes
Comment threadbe/src/olap/delta_writer_v2.cpp Outdated
}
}
SCOPED_RAW_TIMER(&_write_memtable_time);
ThreadCpuStopWatch cpu_time_stop_watch;

@yiguoleiyiguoleiAug 10, 2024

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

我觉得这个得加到async result writer 那个线程那里，这样所有的都被cover 住了

yiguolei
yiguolei reviewed Aug 10, 2024
View reviewed changes
Comment threadfe/fe-core/src/main/java/org/apache/doris/catalog/SchemaTable.java Outdated
builder().column("BE_ID", ScalarType.createType(PrimitiveType.BIGINT))
.column("WORKLOAD_GROUP_ID", ScalarType.createType(PrimitiveType.BIGINT))
.column("MEMORY_USAGE_BYTES", ScalarType.createType(PrimitiveType.BIGINT))
.column("CPU_USAGE", ScalarType.createVarchar(256))

@yiguoleiyiguoleiAug 10, 2024

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

要不把这列的类型定义为整数，这样用户可以排序。
然后把列名改成CPU_USAGE(%)这样

yiguolei
yiguolei reviewed Aug 10, 2024
View reviewed changes
Comment threadbe/src/runtime/workload_management/io_throttle.h
yiguolei
yiguolei reviewed Aug 10, 2024
View reviewed changes
Comment threadbe/src/runtime/thread_context.h
yiguolei
yiguolei reviewed Aug 10, 2024
View reviewed changes
Comment threadbe/src/runtime/workload_management/io_throttle.cpp Outdated
_next_io_time_micros += ret < 1 ? static_cast<int64_t>(1) : static_cast<int64_t>(ret);
}
_next_io_time_micros += ret < 1 ? static_cast<int64_t>(1) : static_cast<int64_t>(ret);
(*_io_adder) << io_bytes;

@yiguoleiyiguoleiAug 10, 2024

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

这里我感觉local 的也统计了，那么之前的就没必要再那个宏里修改了，直接都用这里的统计

github-actions[bot]
github-actionsBot reviewed Aug 10, 2024
View reviewed changes

@github-actionsgithub-actionsBot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

clang-tidy made some suggestions

Comment threadbe/src/runtime/workload_management/io_throttle.cpp
github-actions[bot]
github-actionsBot reviewed Aug 10, 2024
View reviewed changes

@github-actionsgithub-actionsBot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

clang-tidy made some suggestions

Comment threadbe/src/runtime/workload_group/workload_group_manager.cpp
Comment threadbe/src/runtime/workload_management/io_throttle.cpp
Comment threadbe/src/runtime/workload_management/io_throttle.cpp
@wangbo
wangboforce-pushed the 0809_wg_monitor branch 2 times, most recently from 7725bb5 to f3f42f0CompareAugust 10, 2024 07:34
@github-actionsgithub-actionsBot added the approved Indicates a PR has been approved by one committer. label Aug 10, 2024
@github-actions

github-actionsBot commented Aug 10, 2024

Copy link
Copy Markdown
Contributor

PR approved by at least one committer and no changes requested.

@github-actions

github-actionsBot commented Aug 10, 2024

Copy link
Copy Markdown
Contributor

PR approved by anyone and no changes requested.

@yiguolei

yiguolei commented Aug 10, 2024

Copy link
Copy Markdown
Contributor

run buildall

@doris-robot

doris-robot commented Aug 10, 2024

Copy link
Copy Markdown
TPC-H: Total hot run time: 39938 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 3bffef30f6b06fea87b0b810e4d64481cb16f6a6, data reload: false
------ Round 1 ----------------------------------
q1	17710	4665	4397	4397
q2	2022	177	171	171
q3	10524	1261	1173	1173
q4	10163	788	692	692
q5	7493	2524	2489	2489
q6	226	138	144	138
q7	990	600	614	600
q8	9318	1929	1952	1929
q9	8759	6632	6572	6572
q10	7074	2194	2214	2194
q11	495	253	248	248
q12	398	229	229	229
q13	17766	3017	3037	3017
q14	290	243	241	241
q15	536	503	491	491
q16	514	392	394	392
q17	984	722	682	682
q18	8281	7610	7451	7451
q19	4464	978	1072	978
q20	746	343	325	325
q21	5393	4710	4512	4512
q22	1130	1047	1017	1017
Total cold run time: 115276 ms
Total hot run time: 39938 ms
----- Round 2, with runtime_filter_mode=off -----
q1	4547	4291	4325	4291
q2	383	276	274	274
q3	2845	2672	2769	2672
q4	2053	1804	1726	1726
q5	5671	5563	5507	5507
q6	226	136	139	136
q7	2136	1813	1795	1795
q8	3326	3471	3522	3471
q9	8808	8815	8852	8815
q10	3551	3317	3170	3170
q11	607	502	493	493
q12	841	663	660	660
q13	16571	3173	3181	3173
q14	323	301	290	290
q15	538	516	512	512
q16	506	432	434	432
q17	1847	1527	1554	1527
q18	8183	8020	7927	7927
q19	1788	1612	1658	1612
q20	2275	1919	1896	1896
q21	8677	5523	5274	5274
q22	1155	1068	1061	1061
Total cold run time: 76857 ms
Total hot run time: 56714 ms

@doris-robot

doris-robot commented Aug 10, 2024

Copy link
Copy Markdown
TPC-DS: Total hot run time: 204326 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 3bffef30f6b06fea87b0b810e4d64481cb16f6a6, data reload: false
query1	908	376	371	371
query2	6456	1935	1874	1874
query3	6631	210	217	210
query4	34204	23137	23079	23079
query5	3651	501	518	501
query6	284	175	174	174
query7	4593	298	300	298
query8	249	209	205	205
query9	8555	2519	2467	2467
query10	545	470	432	432
query11	17409	14898	15208	14898
query12	137	123	98	98
query13	1631	372	359	359
query14	10268	7704	7881	7704
query15	262	206	256	206
query16	7410	477	490	477
query17	1312	579	559	559
query18	1709	295	294	294
query19	218	143	152	143
query20	125	117	115	115
query21	209	105	110	105
query22	4504	4315	4451	4315
query23	34400	33638	33786	33638
query24	11340	2668	2628	2628
query25	549	375	371	371
query26	702	155	151	151
query27	2174	282	276	276
query28	5885	2042	2025	2025
query29	777	414	411	411
query30	255	152	150	150
query31	980	760	723	723
query32	100	56	60	56
query33	626	281	283	281
query34	861	470	482	470
query35	962	858	843	843
query36	1087	927	939	927
query37	143	79	83	79
query38	4280	4232	4213	4213
query39	1455	1410	1395	1395
query40	206	124	121	121
query41	52	46	46	46
query42	119	101	98	98
query43	555	481	490	481
query44	1117	774	727	727
query45	238	205	202	202
query46	1099	743	795	743
query47	1848	1747	1752	1747
query48	373	298	303	298
query49	837	417	432	417
query50	805	415	421	415
query51	6778	6597	6564	6564
query52	112	95	95	95
query53	263	182	192	182
query54	892	456	452	452
query55	75	78	74	74
query56	276	242	245	242
query57	1162	1048	1081	1048
query58	239	232	253	232
query59	2953	2762	2922	2762
query60	299	267	266	266
query61	101	110	132	110
query62	792	640	637	637
query63	211	186	181	181
query64	9192	2256	1736	1736
query65	3231	3152	3138	3138
query66	723	336	329	329
query67	15192	14863	14698	14698
query68	4477	550	565	550
query69	445	385	383	383
query70	1203	1181	1153	1153
query71	377	293	284	284
query72	19530	18097	17891	17891
query73	767	330	329	329
query74	8973	8802	8829	8802
query75	3343	2651	2702	2651
query76	2156	1021	984	984
query77	521	318	325	318
query78	11211	10016	9573	9573
query79	2824	532	529	529
query80	2062	509	505	505
query81	627	225	231	225
query82	661	144	137	137
query83	290	147	147	147
query84	274	80	80	80
query85	1390	292	273	273
query86	432	296	311	296
query87	4805	4617	4609	4609
query88	3720	2502	2468	2468
query89	408	305	288	288
query90	1926	199	203	199
query91	125	95	95	95
query92	66	53	51	51
query93	3924	545	538	538
query94	842	301	297	297
query95	417	269	263	263
query96	620	276	271	271
query97	3235	3091	3139	3091
query98	218	196	191	191
query99	1592	1293	1299	1293
Total cold run time: 311047 ms
Total hot run time: 204326 ms

@doris-robot

doris-robot commented Aug 10, 2024

Copy link
Copy Markdown
ClickBench: Total hot run time: 30.93 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 3bffef30f6b06fea87b0b810e4d64481cb16f6a6, data reload: false
query1	0.04	0.04	0.04
query2	0.07	0.04	0.04
query3	0.23	0.05	0.06
query4	1.67	0.09	0.08
query5	0.48	0.48	0.50
query6	1.14	0.74	0.73
query7	0.02	0.01	0.02
query8	0.05	0.05	0.05
query9	0.56	0.49	0.49
query10	0.54	0.54	0.54
query11	0.15	0.12	0.12
query12	0.15	0.12	0.12
query13	0.62	0.60	0.58
query14	0.75	0.78	0.78
query15	0.86	0.82	0.82
query16	0.35	0.36	0.36
query17	1.01	0.97	0.96
query18	0.24	0.23	0.22
query19	1.77	1.69	1.72
query20	0.01	0.00	0.01
query21	15.45	0.78	0.66
query22	4.05	7.48	2.20
query23	18.25	1.31	1.24
query24	2.06	0.24	0.21
query25	0.14	0.07	0.08
query26	0.30	0.22	0.22
query27	0.46	0.23	0.23
query28	13.33	1.03	1.00
query29	12.63	3.32	3.30
query30	0.24	0.05	0.06
query31	2.87	0.40	0.38
query32	3.28	0.50	0.48
query33	2.90	2.91	2.96
query34	16.87	4.36	4.34
query35	4.37	4.40	4.40
query36	0.65	0.48	0.48
query37	0.18	0.15	0.15
query38	0.16	0.15	0.15
query39	0.04	0.04	0.04
query40	0.14	0.12	0.13
query41	0.09	0.06	0.05
query42	0.06	0.05	0.05
query43	0.05	0.04	0.04
Total cold run time: 109.28 s
Total hot run time: 30.93 s

zclllyybb
zclllyybb approved these changes Aug 11, 2024
View reviewed changes

@zclllyybbzclllyybb left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. please add apache/doris-website's pr link in your pr content.

@yiguolei
yiguolei merged commit c77bca3 into apache:masterAug 12, 2024
wyxxxcat pushed a commit to wyxxxcat/doris that referenced this pull request Aug 14, 2024
@wangbo@wyxxxcat
[Improment]Add workload group resource usage (apache#39177)
75bb34d
## Proposed changes
```
mysql [information_schema]>select BE_ID,WORKLOAD_GROUP_ID,CPU_USAGE,LOCAL_SCAN_BYTES_PER_SECOND/1024/1024 as scan_io_mb, MEMORY_USAGE_BYTES/1024/1024 mem_mb from workload_group_resource_usage;
+-------+-------------------+-----------+--------------------+-------------------+
| BE_ID | WORKLOAD_GROUP_ID | CPU_USAGE | scan_io_mb | mem_mb |
+-------+-------------------+-----------+--------------------+-------------------+
| 10005 | 62053 | 61.41% | 1516.4589414596558 | 27970.84313774109 |
| 10005 | 1 | 0.00% | 0 | 0 |
+-------+-------------------+-----------+--------------------+-------------------+
```
dataroaring pushed a commit that referenced this pull request Aug 17, 2024
@wangbo@dataroaring
[Improment]Add workload group resource usage (#39177)
d2c3790
## Proposed changes
```
mysql [information_schema]>select BE_ID,WORKLOAD_GROUP_ID,CPU_USAGE,LOCAL_SCAN_BYTES_PER_SECOND/1024/1024 as scan_io_mb, MEMORY_USAGE_BYTES/1024/1024 mem_mb from workload_group_resource_usage;
+-------+-------------------+-----------+--------------------+-------------------+
| BE_ID | WORKLOAD_GROUP_ID | CPU_USAGE | scan_io_mb | mem_mb |
+-------+-------------------+-----------+--------------------+-------------------+
| 10005 | 62053 | 61.41% | 1516.4589414596558 | 27970.84313774109 |
| 10005 | 1 | 0.00% | 0 | 0 |
+-------+-------------------+-----------+--------------------+-------------------+
```
wangbo added a commit to wangbo/incubator-doris that referenced this pull request Aug 23, 2024
@wangbo
[Improment]Add workload group resource usage (apache#39177)
fc16413
```
mysql [information_schema]>select BE_ID,WORKLOAD_GROUP_ID,CPU_USAGE,LOCAL_SCAN_BYTES_PER_SECOND/1024/1024 as scan_io_mb, MEMORY_USAGE_BYTES/1024/1024 mem_mb from workload_group_resource_usage;
+-------+-------------------+-----------+--------------------+-------------------+
| BE_ID | WORKLOAD_GROUP_ID | CPU_USAGE | scan_io_mb | mem_mb |
+-------+-------------------+-----------+--------------------+-------------------+
| 10005 | 62053 | 61.41% | 1516.4589414596558 | 27970.84313774109 |
| 10005 | 1 | 0.00% | 0 | 0 |
+-------+-------------------+-----------+--------------------+-------------------+
```
wangbo added a commit to wangbo/incubator-doris that referenced this pull request Aug 23, 2024
@wangbo
[Improment]Add workload group resource usage (apache#39177)
5f83881
```
mysql [information_schema]>select BE_ID,WORKLOAD_GROUP_ID,CPU_USAGE,LOCAL_SCAN_BYTES_PER_SECOND/1024/1024 as scan_io_mb, MEMORY_USAGE_BYTES/1024/1024 mem_mb from workload_group_resource_usage;
+-------+-------------------+-----------+--------------------+-------------------+
| BE_ID | WORKLOAD_GROUP_ID | CPU_USAGE | scan_io_mb | mem_mb |
+-------+-------------------+-----------+--------------------+-------------------+
| 10005 | 62053 | 61.41% | 1516.4589414596558 | 27970.84313774109 |
| 10005 | 1 | 0.00% | 0 | 0 |
+-------+-------------------+-----------+--------------------+-------------------+
```
@gavinchougavinchou mentioned this pull request Oct 13, 2024
Closed
HappenLee pushed a commit to HappenLee/incubator-doris that referenced this pull request Apr 24, 2026
@wangbo@HappenLee
[Improment]Add workload group resource usage (apache#39177)
b3db8d7
## Proposed changes
```
mysql [information_schema]>select BE_ID,WORKLOAD_GROUP_ID,CPU_USAGE,LOCAL_SCAN_BYTES_PER_SECOND/1024/1024 as scan_io_mb, MEMORY_USAGE_BYTES/1024/1024 mem_mb from workload_group_resource_usage;
+-------+-------------------+-----------+--------------------+-------------------+
| BE_ID | WORKLOAD_GROUP_ID | CPU_USAGE | scan_io_mb | mem_mb |
+-------+-------------------+-----------+--------------------+-------------------+
| 10005 | 62053 | 61.41% | 1516.4589414596558 | 27970.84313774109 |
| 10005 | 1 | 0.00% | 0 | 0 |
+-------+-------------------+-----------+--------------------+-------------------+
```
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@github-actionsgithub-actions[bot]github-actions[bot] left review comments
@yiguoleiyiguoleiyiguolei approved these changes
@zclllyybbzclllyybbzclllyybb approved these changes

Assignees

No one assigned

Labels

approvedIndicates a PR has been approved by one committer.dev/3.0.2-mergeddoingmeta-changereviewed

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@wangbo@doris-robot@yiguolei@zclllyybb@dataroaring