[feature](statistics) Statistics derivation.Step 1:ScanNode implement… by zhengshiJ · Pull Request #8947 · apache/doris

zhengshiJ · 2022-04-11T02:09:59Z

The purpose of this function is to modify the derivation of the current statistics.
There are two modifications:

The statistical information derivation is removed from the Node node.
Modify the current way of obtaining cardinality, and change it to obtain it from statistical information.
For details on the modification method, see: https://shimo.im/docs/473QyOzWBjHMlZ3w
The implementation of this function is divided into multiple submissions, and this submission implements the derivation of scan_node statistics.

Proposed changes

Issue Number: close#9241

Problem Summary:

The derivation of the current statistical information is coupled in the node node, and the method of calculating the cardinality is carried out by the mock method, and the ndv value is not obtained correctly.

Checklist(Required)

Does it affect the original behavior: (No)
Has unit tests been added: (No)
Has document been added or modified: (No Need)
Does it need to update dependencies: (No)
Are there any changes that cannot be rolled back: (No)

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

EmmyMiao87 · 2022-04-22T08:46:33Z

+
+ // Currently it simply adds the number of rows of children
+ protected long deriveRowCount() {
+ applyConjunctsSelectivity();


Suggested change
applyConjunctsSelectivity();
rowcount = childrenmax(rowcount)
applyConjunctsSelectivity();

EmmyMiao87

LGTM

github-actions · 2022-04-25T04:44:05Z

PR approved by at least one committer and no changes requested.

github-actions · 2022-04-25T04:44:07Z

PR approved by anyone and no changes requested.

EmmyMiao87 · 2022-04-25T04:45:51Z

The FE ut is failed case:

AlterViewStmtTest.testNormal
TableQueryPlanActionTest.testQueryPlanAction

Those case will be fixed in PR: #9158

The BE unit test problem is independent of the current pr.

EmmyMiao87

LGTM

github-actions · 2022-04-26T10:38:27Z

PR approved by at least one committer and no changes requested.

EmmyMiao87

The code base line is incorrect

…ation

…blem

EmmyMiao87

LGTM

github-actions · 2022-04-28T02:51:52Z

PR approved by at least one committer and no changes requested.

apache#8947) * [feature](statistics) Statistics derivation.Step 1:ScanNode implementation Co-authored-by: jianghaochen <jianghaochen@meituan.com>

github-actionsBot added the area/planner Issues or PRs related to the query planner label Apr 11, 2022

zhengshiJ force-pushed the master branch from be95006 to 9c0061eCompare April 11, 2022 02:46

zbtzbtzbt reviewed Apr 11, 2022
View reviewed changes

Comment threadfe/fe-core/src/main/java/org/apache/doris/planner/OlapScanNode.java Outdated

EmmyMiao87 added the area/statistics Issues or PRS related to statistics label Apr 13, 2022

EmmyMiao87 self-assigned this Apr 13, 2022

EmmyMiao87 reviewed Apr 14, 2022
View reviewed changes

Comment threadfe/fe-core/src/main/java/org/apache/doris/statistics/StatsRecursiveDerive.java Outdated
Comment threadfe/fe-core/src/main/java/org/apache/doris/statistics/StatsRecursiveDerive.java Outdated

EmmyMiao87 reviewed Apr 14, 2022
View reviewed changes

Comment threadfe/fe-core/src/main/java/org/apache/doris/planner/SingleNodePlanner.java Outdated
Comment threadfe/fe-core/src/main/java/org/apache/doris/planner/PlanNode.java Outdated
Comment threadfe/fe-core/src/main/java/org/apache/doris/planner/PlanNode.java Outdated

EmmyMiao87 reviewed Apr 14, 2022
View reviewed changes

Comment threadfe/fe-core/src/main/java/org/apache/doris/planner/OlapScanNode.java

zhengshiJ force-pushed the master branch from 46e8892 to 8d231ffCompare April 15, 2022 04:29

EmmyMiao87 reviewed Apr 20, 2022
View reviewed changes

zhengshiJ force-pushed the master branch 2 times, most recently from 66972cf to 9c2ba3aCompare April 20, 2022 06:38

github-actionsBot added the area/load Issues or PRs related to all kinds of load label Apr 20, 2022

EmmyMiao87 reviewed Apr 20, 2022
View reviewed changes

zhengshiJ force-pushed the master branch from 24fb1e5 to 9d7cd85Compare April 21, 2022 06:49

EmmyMiao87 reviewed Apr 22, 2022
View reviewed changes

zhengshiJ force-pushed the master branch from 4f7a5e7 to 26f6032Compare April 22, 2022 10:39

EmmyMiao87 reviewed Apr 22, 2022
View reviewed changes

Comment threadfe/fe-core/src/main/java/org/apache/doris/statistics/BaseStatsDerive.java

zhengshiJ force-pushed the master branch from 26f6032 to 789ab67Compare April 24, 2022 02:06

EmmyMiao87 previously approved these changes Apr 25, 2022
View reviewed changes

github-actionsBot added the approved Indicates a PR has been approved by one committer. label Apr 25, 2022

github-actionsBot added the reviewed label Apr 25, 2022

zhengshiJ dismissed EmmyMiao87’s stale review via 6f8b0dcApril 26, 2022 02:32

github-actionsBot removed the approved Indicates a PR has been approved by one committer. label Apr 26, 2022

EmmyMiao87 approved these changes Apr 26, 2022
View reviewed changes

github-actionsBot added the approved Indicates a PR has been approved by one committer. label Apr 26, 2022

EmmyMiao87 requested changes Apr 26, 2022
View reviewed changes

github-actionsBot removed the approved Indicates a PR has been approved by one committer. label Apr 26, 2022

zhengshiJ force-pushed the master branch from 4735652 to f56fcdaCompare April 27, 2022 03:42

github-actionsBot added area/routine load area/spark-load Issues or PRs related to the spark load area/sql/function Issues or PRs related to the SQL functions area/vectorization kind/docs Categorizes issue or PR as related to documentation. kind/test labels Apr 27, 2022

zhengshiJ force-pushed the master branch from f56fcda to 789ab67Compare April 27, 2022 03:56

jianghaochen added 3 commits April 27, 2022 12:12

[feature](statistics) Statistics derivation.Step 1:ScanNode implement…
ee85f1c
…ation

Fix problems in review
9fbbece

Modify the review question
1e4da1c

zhengshiJ force-pushed the master branch from 789ab67 to fb4927eCompare April 27, 2022 04:14

zhengshiJ mentioned this pull request Apr 27, 2022
[feature](statistics) Statistics derivation.Step 1:ScanNode implement… #9258
Closed

Fix the ut problem introduced byapache#9047and fixed the review pro…
25bd407
…blem

zhengshiJ force-pushed the master branch from 95d5d31 to 25bd407Compare April 27, 2022 12:47

EmmyMiao87 approved these changes Apr 28, 2022
View reviewed changes

github-actionsBot added the approved Indicates a PR has been approved by one committer. label Apr 28, 2022

yiguolei merged commit 9ef09b8 into apache:masterApr 29, 2022

morningman added the dev/backlog waiting to be merged in future dev branch label Apr 29, 2022

	applyConjunctsSelectivity();
	rowcount = childrenmax(rowcount)
	applyConjunctsSelectivity();

Uh oh!

Conversation

zhengshiJ commented Apr 11, 2022• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Proposed changes

Problem Summary:

Checklist(Required)

Further comments

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

EmmyMiao87Apr 22, 2022

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

EmmyMiao87 left a comment

Choose a reason for hiding this comment

Uh oh!

github-actionsBot commented Apr 25, 2022

Uh oh!

github-actionsBot commented Apr 25, 2022

Uh oh!

EmmyMiao87 commented Apr 25, 2022

Uh oh!

EmmyMiao87 left a comment

Choose a reason for hiding this comment

Uh oh!

github-actionsBot commented Apr 26, 2022

Uh oh!

EmmyMiao87 left a comment

Choose a reason for hiding this comment

Uh oh!

EmmyMiao87 left a comment

Choose a reason for hiding this comment

Uh oh!

github-actionsBot commented Apr 28, 2022

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

zhengshiJ commented Apr 11, 2022•
edited
Loading