[feature](profilev2) Preliminary support for profilev2. by Mryange · Pull Request #24881 · apache/doris

Mryange · 2023-09-25T12:20:54Z

Proposed changes

You can set the level of counters on the backend using ADD_COUNTER_WITH_LEVEL/ADD_TIMER_WITH_LEVEL. The profile can then merge counters with level 1.
set profile_level = 1;
such as
sql
select count(*) from customer join item on c_customer_sk = i_item_sk

profile

Simple profile PLAN FRAGMENT 0
OUTPUT EXPRS:
count(*)
PARTITION: UNPARTITIONED
VRESULT SINK
MYSQL_PROTOCAL
7:VAGGREGATE (merge finalize)
| output: count(partial_count(*))[#44]
| group by: | cardinality=1
| TotalTime: avg 725.608us, max 725.608us, min 725.608us
| RowsReturned: 1
| 6:VEXCHANGE
offset: 0
TotalTime: avg 52.411us, max 52.411us, min 52.411us
RowsReturned: 8
PLAN FRAGMENT 1
PARTITION: HASH_PARTITIONED: c_customer_sk
STREAM DATA SINK
EXCHANGE ID: 06
UNPARTITIONED
TotalTime: avg 106.263us, max 118.38us, min 81.403us
BlocksSent: 8
5:VAGGREGATE (update serialize)
| output: partial_count(*)[#43]
| group by: | cardinality=1
| TotalTime: avg 679.296us, max 739.395us, min 554.904us
| BuildTime: avg 33.198us, max 48.387us, min 28.880us
| ExecTime: avg 27.633us, max 40.278us, min 24.537us
| RowsReturned: 8
| 4:VHASH JOIN
| join op: INNER JOIN(PARTITIONED)[]
| equal join conjunct: c_customer_sk = i_item_sk
| runtime filters: RF000[bloom] <- i_item_sk(18000/16384/1048576)
| cardinality=17,740
| vec output tuple id: 3
| vIntermediate tuple ids: 2 | hash output slot ids: 22 | RowsReturned: 18.0K (18000)
| ProbeRows: 18.0K (18000)
| ProbeTime: avg 862.308us, max 1.576ms, min 666.28us
| BuildRows: 18.0K (18000)
| BuildTime: avg 3.8ms, max 3.860ms, min 2.317ms
| |----1:VEXCHANGE
| offset: 0
| TotalTime: avg 48.822us, max 67.459us, min 30.380us
| RowsReturned: 18.0K (18000)
| 3:VEXCHANGE
offset: 0
TotalTime: avg 33.162us, max 39.480us, min 28.854us
RowsReturned: 18.0K (18000)
PLAN FRAGMENT 2
PARTITION: HASH_PARTITIONED: c_customer_id
STREAM DATA SINK
EXCHANGE ID: 03
HASH_PARTITIONED: c_customer_sk
TotalTime: avg 753.954us, max 1.210ms, min 499.470us
BlocksSent: 64
2:VOlapScanNode
TABLE: default_cluster:tpcds.customer(customer), PREAGGREGATION: ON
runtime filters: RF000[bloom] -> c_customer_sk
partitions=1/1, tablets=12/12, tabletList=1550745,1550747,1550749 ...
cardinality=100000, avgRowSize=0.0, numNodes=1
pushAggOp=NONE
TotalTime: avg 18.417us, max 41.319us, min 10.189us
RowsReturned: 18.0K (18000)

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

Mryange · 2023-09-25T12:21:13Z

run buildall

github-actions

clang-tidy made some suggestions

github-actions · 2023-09-25T12:28:45Z

 DCHECK(parent_counter_name == ROOT_COUNTER ||
 _counter_map.find(parent_counter_name) != _counter_map.end());
- Counter* counter = _pool->add(new Counter(type, 0));
+ Counter* counter = _pool->add(new Counter(type, 0, level));


warning: variable 'counter' is not initialized [cppcoreguidelines-init-variables]
Suggested change
Counter* counter = _pool->add(new Counter(type, 0, level));
Counter* counter = nullptr = _pool->add(new Counter(type, 0, level));

github-actions · 2023-09-25T12:28:45Z

+ _metadata = md;
+ }
+
+ bool is_set_metadata() const { return _is_set_metadata; }


warning: function 'is_set_metadata' should be marked [[nodiscard]] [modernize-use-nodiscard]
Suggested change
boolis_set_metadata() const { return_is_set_metadata; }
[[nodiscard]] boolis_set_metadata() const { return_is_set_metadata; }

doris-robot · 2023-09-25T13:21:32Z

TeamCity be ut coverage result:
Function Coverage: 36.33% (8115/22339)
Line Coverage: 28.45% (64914/228168)
Region Coverage: 27.34% (33660/123137)
Branch Coverage: 23.99% (17183/71624)
Coverage Report: http://coverage.selectdb-in.cc/coverage/68ac0dc47efd796038ae36ef3ed27a4e5804ad24_68ac0dc47efd796038ae36ef3ed27a4e5804ad24/report/index.html

doris-robot · 2023-09-25T13:25:22Z

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 45.87 seconds
stream load tsv: 558 seconds loaded 74807831229 Bytes, about 127 MB/s
stream load json: 21 seconds loaded 2358488459 Bytes, about 107 MB/s
stream load orc: 65 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 31 seconds loaded 861443392 Bytes, about 26 MB/s
insert into select: 29.0 seconds inserted 10000000 Rows, about 344K ops/s
storage size: 17162099372 Bytes

yiguolei · 2023-09-26T06:16:33Z

 1: required string name
 2: required Metrics.TUnit type
 3: required i64 value 
+ 4: required i64 level


should always be optional, never use required.

yiguolei · 2023-09-26T08:04:37Z

 return add_counter(name, type, "");
 }

+ Counter* add_counter_with_levle(const std::string& name, TUnit::type type, int64_t level) {


Mryange · 2023-09-26T08:33:19Z

run buildall

doris-robot · 2023-09-26T09:31:21Z

TeamCity be ut coverage result:
Function Coverage: 36.44% (8160/22391)
Line Coverage: 28.57% (65378/228874)
Region Coverage: 27.42% (33869/123518)
Branch Coverage: 24.06% (17279/71830)
Coverage Report: http://coverage.selectdb-in.cc/coverage/42b2aaf602c5b0a88de50cfeba38a6b5cb035db8_42b2aaf602c5b0a88de50cfeba38a6b5cb035db8/report/index.html

doris-robot · 2023-09-26T09:34:58Z

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 48.95 seconds
stream load tsv: 565 seconds loaded 74807831229 Bytes, about 126 MB/s
stream load json: 21 seconds loaded 2358488459 Bytes, about 107 MB/s
stream load orc: 66 seconds loaded 1101869774 Bytes, about 15 MB/s
stream load parquet: 34 seconds loaded 861443392 Bytes, about 24 MB/s
insert into select: 29.2 seconds inserted 10000000 Rows, about 342K ops/s
storage size: 17162276507 Bytes

yiguolei · 2023-09-26T11:15:13Z

 output.append(prefix).append("SHORT-CIRCUIT");
 }
-
 return output.toString();


not modify useless file

yiguolei · 2023-09-26T11:18:48Z

@@ -534,13 +508,18 @@ public String toString() {
 }

 public String getSimpleString() {


rename to getLevel1Profile

yiguolei · 2023-09-26T11:20:37Z

 return;
 }
+ if (node.isSetMetadata()) {
+ this.nodeid = (int) node.getMetadata();


I think we should not depend on it.
Add a specific field in profile object named plan_node_id.
metadata is too normal.

Mryange · 2023-09-26T12:01:20Z

run buildall

doris-robot · 2023-09-26T12:57:58Z

TeamCity be ut coverage result:
Function Coverage: 36.34% (8142/22406)
Line Coverage: 28.46% (65149/228887)
Region Coverage: 27.42% (33783/123200)
Branch Coverage: 24.08% (17246/71624)
Coverage Report: http://coverage.selectdb-in.cc/coverage/d15776560fa5a24ee9ebb3cb95458e7affb1e350_d15776560fa5a24ee9ebb3cb95458e7affb1e350/report/index.html

doris-robot · 2023-09-26T13:08:29Z

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 45.79 seconds
stream load tsv: 557 seconds loaded 74807831229 Bytes, about 128 MB/s
stream load json: 20 seconds loaded 2358488459 Bytes, about 112 MB/s
stream load orc: 65 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 28.9 seconds inserted 10000000 Rows, about 346K ops/s
storage size: 17161993743 Bytes

Mryange · 2023-09-26T14:51:27Z

run buildall

doris-robot · 2023-09-26T15:38:09Z

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 45.3 seconds
stream load tsv: 559 seconds loaded 74807831229 Bytes, about 127 MB/s
stream load json: 21 seconds loaded 2358488459 Bytes, about 107 MB/s
stream load orc: 66 seconds loaded 1101869774 Bytes, about 15 MB/s
stream load parquet: 33 seconds loaded 861443392 Bytes, about 24 MB/s
insert into select: 29.0 seconds inserted 10000000 Rows, about 344K ops/s
storage size: 17162175541 Bytes

doris-robot · 2023-09-26T15:55:55Z

TeamCity be ut coverage result:
Function Coverage: 36.34% (8146/22416)
Line Coverage: 28.45% (65176/229109)
Region Coverage: 27.40% (33790/123301)
Branch Coverage: 24.07% (17250/71666)
Coverage Report: http://coverage.selectdb-in.cc/coverage/d15776560fa5a24ee9ebb3cb95458e7affb1e350_d15776560fa5a24ee9ebb3cb95458e7affb1e350/report/index.html

yiguolei · 2023-09-28T02:22:35Z

 std::lock_guard<std::mutex> l(_children_lock);
 DCHECK(_child_map.find(name) == _child_map.end());
 RuntimeProfile* child = _pool->add(new RuntimeProfile(name));
+ if (this->is_set_metadata()) {


这样做可能是错的，因为如果meta data 里是node id，那么child 会有自己的node id

yiguolei · 2023-10-04T09:15:06Z

run buildall

yiguolei · 2023-10-06T08:58:53Z

run buildall

doris-robot · 2023-10-06T10:06:18Z

TeamCity be ut coverage result:
Function Coverage: 36.30% (8150/22450)
Line Coverage: 28.43% (65172/229256)
Region Coverage: 27.36% (33749/123339)
Branch Coverage: 24.04% (17221/71632)
Coverage Report: http://coverage.selectdb-in.cc/coverage/00a7361b68150fa4c7582a8b6f170db019d0ff17_00a7361b68150fa4c7582a8b6f170db019d0ff17/report/index.html

doris-robot · 2023-10-06T10:10:39Z

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 45.37 seconds
stream load tsv: 565 seconds loaded 74807831229 Bytes, about 126 MB/s
stream load json: 21 seconds loaded 2358488459 Bytes, about 107 MB/s
stream load orc: 64 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 29.1 seconds inserted 10000000 Rows, about 343K ops/s
storage size: 17162476842 Bytes

yiguolei

LGTM

github-actions · 2023-10-07T02:10:17Z

PR approved by at least one committer and no changes requested.

github-actions · 2023-10-07T02:10:19Z

PR approved by anyone and no changes requested.

Gabriel39

LGTM

You can set the level of counters on the backend using ADD_COUNTER_WITH_LEVEL/ADD_TIMER_WITH_LEVEL. The profile can then merge counters with level 1. set profile_level = 1; such as sql select count(*) from customer join item on c_customer_sk = i_item_sk profile Simple profile PLAN FRAGMENT 0 OUTPUT EXPRS: count(*) PARTITION: UNPARTITIONED VRESULT SINK MYSQL_PROTOCAL 7:VAGGREGATE (merge finalize) | output: count(partial_count(*))[apache#44] | group by: | cardinality=1 | TotalTime: avg 725.608us, max 725.608us, min 725.608us | RowsReturned: 1 | 6:VEXCHANGE offset: 0 TotalTime: avg 52.411us, max 52.411us, min 52.411us RowsReturned: 8 PLAN FRAGMENT 1 PARTITION: HASH_PARTITIONED: c_customer_sk STREAM DATA SINK EXCHANGE ID: 06 UNPARTITIONED TotalTime: avg 106.263us, max 118.38us, min 81.403us BlocksSent: 8 5:VAGGREGATE (update serialize) | output: partial_count(*)[apache#43] | group by: | cardinality=1 | TotalTime: avg 679.296us, max 739.395us, min 554.904us | BuildTime: avg 33.198us, max 48.387us, min 28.880us | ExecTime: avg 27.633us, max 40.278us, min 24.537us | RowsReturned: 8 | 4:VHASH JOIN | join op: INNER JOIN(PARTITIONED)[] | equal join conjunct: c_customer_sk = i_item_sk | runtime filters: RF000[bloom] <- i_item_sk(18000/16384/1048576) | cardinality=17,740 | vec output tuple id: 3 | vIntermediate tuple ids: 2 | hash output slot ids: 22 | RowsReturned: 18.0K (18000) | ProbeRows: 18.0K (18000) | ProbeTime: avg 862.308us, max 1.576ms, min 666.28us | BuildRows: 18.0K (18000) | BuildTime: avg 3.8ms, max 3.860ms, min 2.317ms | |----1:VEXCHANGE | offset: 0 | TotalTime: avg 48.822us, max 67.459us, min 30.380us | RowsReturned: 18.0K (18000) | 3:VEXCHANGE offset: 0 TotalTime: avg 33.162us, max 39.480us, min 28.854us RowsReturned: 18.0K (18000) PLAN FRAGMENT 2 PARTITION: HASH_PARTITIONED: c_customer_id STREAM DATA SINK EXCHANGE ID: 03 HASH_PARTITIONED: c_customer_sk TotalTime: avg 753.954us, max 1.210ms, min 499.470us BlocksSent: 64 2:VOlapScanNode TABLE: default_cluster:tpcds.customer(customer), PREAGGREGATION: ON runtime filters: RF000[bloom] -> c_customer_sk partitions=1/1, tablets=12/12, tabletList=1550745,1550747,1550749 ... cardinality=100000, avgRowSize=0.0, numNodes=1 pushAggOp=NONE TotalTime: avg 18.417us, max 41.319us, min 10.189us RowsReturned: 18.0K (18000) --------- Co-authored-by: yiguolei <676222867@qq.com>

Mryange added 3 commits September 25, 2023 18:03

update
f7a3f2e

rebase
88d69b1

format
68ac0dc

github-actionsBot reviewed Sep 25, 2023
View reviewed changes

yiguolei added the profilev2 label Sep 26, 2023

yiguolei reviewed Sep 26, 2023
View reviewed changes

use profile level
42b2aaf

yiguolei reviewed Sep 26, 2023
View reviewed changes

format
d157765

yiguolei reviewed Sep 28, 2023
View reviewed changes

Merge branch 'master' into dev-57
fe4bfc1

Merge branch 'master' into dev-57
00a7361

yiguolei approved these changes Oct 7, 2023
View reviewed changes

github-actionsBot added the approved Indicates a PR has been approved by one committer. label Oct 7, 2023

github-actionsBot added the reviewed label Oct 7, 2023

Gabriel39 approved these changes Oct 7, 2023
View reviewed changes

yiguolei merged commit 0631ed6 into apache:masterOct 7, 2023

	Counter* counter = _pool->add(new Counter(type, 0, level));
	Counter* counter = nullptr = _pool->add(new Counter(type, 0, level));

	boolis_set_metadata() const { return_is_set_metadata; }
	[[nodiscard]] boolis_set_metadata() const { return_is_set_metadata; }

		@@ -534,13 +508,18 @@ public String toString() {
		}

		public String getSimpleString() {

Uh oh!

Conversation

Mryange commented Sep 25, 2023• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Proposed changes

Further comments

Uh oh!

Mryange commented Sep 25, 2023

Uh oh!

github-actionsBot left a comment

Choose a reason for hiding this comment

Uh oh!

github-actionsBotSep 25, 2023

Choose a reason for hiding this comment

Uh oh!

github-actionsBotSep 25, 2023

Choose a reason for hiding this comment

Uh oh!

doris-robot commented Sep 25, 2023

Uh oh!

doris-robot commented Sep 25, 2023

Uh oh!

yiguoleiSep 26, 2023

Choose a reason for hiding this comment

Uh oh!

yiguoleiSep 26, 2023

Choose a reason for hiding this comment

Uh oh!

Mryange commented Sep 26, 2023

Uh oh!

doris-robot commented Sep 26, 2023

Uh oh!

doris-robot commented Sep 26, 2023

Uh oh!

yiguoleiSep 26, 2023

Choose a reason for hiding this comment

Uh oh!

yiguoleiSep 26, 2023

Choose a reason for hiding this comment

Uh oh!

yiguoleiSep 26, 2023

Choose a reason for hiding this comment

Uh oh!

Mryange commented Sep 26, 2023

Uh oh!

doris-robot commented Sep 26, 2023

Uh oh!

doris-robot commented Sep 26, 2023

Uh oh!

Mryange commented Sep 26, 2023

Uh oh!

doris-robot commented Sep 26, 2023

Uh oh!

doris-robot commented Sep 26, 2023

Uh oh!

yiguoleiSep 28, 2023

Choose a reason for hiding this comment

Uh oh!

yiguolei commented Oct 4, 2023

Uh oh!

yiguolei commented Oct 6, 2023

Uh oh!

doris-robot commented Oct 6, 2023

Uh oh!

doris-robot commented Oct 6, 2023

Uh oh!

yiguolei left a comment

Choose a reason for hiding this comment

Uh oh!

github-actionsBot commented Oct 7, 2023

Uh oh!

github-actionsBot commented Oct 7, 2023

Uh oh!

Gabriel39 left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Mryange commented Sep 25, 2023•
edited
Loading