Skip to content

[Enhancement](Nereids)Add nereids minidump - #18747

Merged
jackwener merged 1 commit into
apache:masterfrom
LiBinfeng-01:dump_new
May 11, 2023
Merged

[Enhancement](Nereids)Add nereids minidump#18747
jackwener merged 1 commit into
apache:masterfrom
LiBinfeng-01:dump_new

Conversation

@LiBinfeng-01

@LiBinfeng-01LiBinfeng-01 commented Apr 17, 2023

Copy link
Copy Markdown
Contributor

Add runnable mini dump demo in MinidumpTest

Problem summary

Describe your changes.

Checklist(Required)

  • Does it affect the original behavior
  • Has unit tests been added
  • Has document been added or modified
  • Does it need to update dependencies
  • Is this PR support rollback (If NO, please explain WHY)

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

@LiBinfeng-01

LiBinfeng-01 commented Apr 17, 2023

Copy link
Copy Markdown
ContributorAuthor

run build all

@LiBinfeng-01

LiBinfeng-01 commented Apr 17, 2023

Copy link
Copy Markdown
ContributorAuthor

run buildall

@morrySnow

morrySnow commented Apr 24, 2023

Copy link
Copy Markdown
Contributor

run buildall

@LiBinfeng-01

LiBinfeng-01 commented Apr 24, 2023

Copy link
Copy Markdown
ContributorAuthor

run buildall

@hello-stephen

hello-stephen commented Apr 24, 2023•
edited
Loading

Copy link
Copy Markdown
Contributor

TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 33.56 seconds
stream load tsv: 427 seconds loaded 74807831229 Bytes, about 167 MB/s
stream load json: 22 seconds loaded 2358488459 Bytes, about 102 MB/s
stream load orc: 60 seconds loaded 1101869774 Bytes, about 17 MB/s
stream load parquet: 31 seconds loaded 861443392 Bytes, about 26 MB/s
https://doris-community-test-1308700295.cos.ap-hongkong.myqcloud.com/tmp/20230510133955_clickbench_pr_141841.html

@LiBinfeng-01

LiBinfeng-01 commented Apr 25, 2023

Copy link
Copy Markdown
ContributorAuthor

run fest

@LiBinfeng-01

LiBinfeng-01 commented Apr 25, 2023

Copy link
Copy Markdown
ContributorAuthor

run feut

@LiBinfeng-01

LiBinfeng-01 commented Apr 26, 2023

Copy link
Copy Markdown
ContributorAuthor

run feut

@LiBinfeng-01

LiBinfeng-01 commented Apr 26, 2023

Copy link
Copy Markdown
ContributorAuthor

run buildall

@LiBinfeng-01

LiBinfeng-01 commented Apr 27, 2023

Copy link
Copy Markdown
ContributorAuthor

run feut

@LiBinfeng-01

LiBinfeng-01 commented Apr 27, 2023

Copy link
Copy Markdown
ContributorAuthor

run feut

@LiBinfeng-01

LiBinfeng-01 commented Apr 27, 2023

Copy link
Copy Markdown
ContributorAuthor

run buildall

jackwener
jackwener reviewed Apr 27, 2023
View reviewed changes
Comment threadfe/fe-core/src/main/java/org/apache/doris/nereids/CascadesContext.java Outdated

/** get table by table name, try to get from information from dumpfile first */
public Table getTableByName(String tableName) {
assert (tables != null);

@jackwenerjackwenerApr 27, 2023

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

use Precondition.checkState()

@LiBinfeng-01LiBinfeng-01Apr 27, 2023

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

get

@LiBinfeng-01LiBinfeng-01Apr 27, 2023

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done

@LiBinfeng-01

LiBinfeng-01 commented Apr 27, 2023

Copy link
Copy Markdown
ContributorAuthor

run buildall

LiBinfeng-01
LiBinfeng-01 commented Apr 27, 2023
View reviewed changes
Comment thread...e-core/src/main/java/org/apache/doris/nereids/trees/plans/physical/AbstractPhysicalJoin.java
Comment on lines +188 to +189
@Override
public JSONObject toJson() {

@LiBinfeng-01LiBinfeng-01Apr 27, 2023

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Because abstractPhysicalJoin can be different with abstractLogicalJoin, so her can have some different implementation

jackwener
jackwener reviewed Apr 27, 2023
View reviewed changes
Comment threadfe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/logical/LogicalBinary.java Outdated
Comment on lines +50 to +60
@Override
public JSONObject toJson() {
JSONObject json = new JSONObject();
json.put("PlanType", getType().toString());
JSONArray childrenJson = new JSONArray();
for (Plan child : children) {
childrenJson.put(child.toJson());
}
json.put("children", childrenJson);
return json;
}

@jackwenerjackwenerApr 27, 2023

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we don't implement this func for UnaryBinary.
We can implement it for AbstractPlan, and check children().size is enough

@LiBinfeng-01LiBinfeng-01Apr 27, 2023•
edited
Loading

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I considered this implementation when start doing this part, but found leafPlan do not have children member, so this can not be done

@LiBinfeng-01

LiBinfeng-01 commented Apr 27, 2023

Copy link
Copy Markdown
ContributorAuthor

run buildall

jackwener
jackwener reviewed Apr 27, 2023
View reviewed changes
Comment threadfe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/Plan.java Outdated
jackwener
jackwener reviewed Apr 27, 2023
View reviewed changes
Comment threadfe/fe-core/src/main/java/org/apache/doris/qe/SessionVariable.java Outdated
Comment on lines +945 to +948
public void setEnableNereidsTimeout(boolean enableNereidsTimeout) {
this.enableNereidsTimeout = enableNereidsTimeout;
}

@jackwenerjackwenerApr 27, 2023

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Because variable is public, we don't need get/set

@LiBinfeng-01LiBinfeng-01Apr 27, 2023

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done

jackwener
jackwener reviewed Apr 27, 2023
View reviewed changes
Comment threadfe/fe-core/src/main/java/org/apache/doris/nereids/NereidsPlanner.java
jackwener
jackwener reviewed Apr 27, 2023
View reviewed changes
Comment threadfe/fe-core/src/main/java/org/apache/doris/nereids/minidump/Minidump.java Outdated
Comment on lines +88 to +90
public Minidump() {

}

@jackwenerjackwenerApr 27, 2023

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why add this a empty constructor, it's easy to use wrong by others.

@LiBinfeng-01LiBinfeng-01Apr 27, 2023

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

sure, it was added default by ide, I forgot to remove it

jackwener
jackwener reviewed May 10, 2023
View reviewed changes
Comment threadfe/fe-core/src/main/java/org/apache/doris/qe/ConnectContext.java
jackwener
jackwener reviewed May 10, 2023
View reviewed changes
Comment threadfe/fe-core/src/main/java/org/apache/doris/nereids/jobs/rewrite/CustomRewriteJob.java Outdated
Comment on lines +48 to +51
public RuleType getRuleType() {
return ruleType;
}

@jackwenerjackwenerMay 10, 2023

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
publicRuleTypegetRuleType() {
returnruleType;
}

It's redundant

@LiBinfeng-01LiBinfeng-01May 10, 2023

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

get

@LiBinfeng-01LiBinfeng-01May 10, 2023

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done

@github-actionsgithub-actionsBot removed the approved Indicates a PR has been approved by one committer. label May 10, 2023
jackwener
jackwener reviewed May 10, 2023
View reviewed changes
Comment threadfe/fe-core/src/main/java/org/apache/doris/nereids/minidump/MinidumpUtils.java Outdated
jackwener
jackwener reviewed May 10, 2023
View reviewed changes
Comment threadfe/fe-core/src/main/java/org/apache/doris/nereids/minidump/NereidsTracer.java Outdated
jackwener
jackwener reviewed May 10, 2023
View reviewed changes
Comment threadfe/fe-core/src/main/java/org/apache/doris/nereids/minidump/NereidsTracer.java Outdated
jackwener
jackwener reviewed May 10, 2023
View reviewed changes
Comment threadfe/fe-core/src/main/java/org/apache/doris/nereids/minidump/MinidumpUtils.java Outdated
@LiBinfeng-01

LiBinfeng-01 commented May 10, 2023

Copy link
Copy Markdown
ContributorAuthor

run buildall

@LiBinfeng-01

LiBinfeng-01 commented May 10, 2023

Copy link
Copy Markdown
ContributorAuthor

run buildall

1 similar comment
@LiBinfeng-01

LiBinfeng-01 commented May 10, 2023

Copy link
Copy Markdown
ContributorAuthor

run buildall

@LiBinfeng-01

LiBinfeng-01 commented May 10, 2023

Copy link
Copy Markdown
ContributorAuthor

run buildall

@LiBinfeng-01

LiBinfeng-01 commented May 11, 2023

Copy link
Copy Markdown
ContributorAuthor

run p0

@jackwener
jackwener merged commit 99cef84 into apache:masterMay 11, 2023
morrySnow added a commit to morrySnow/incubator-doris that referenced this pull request May 29, 2024
@morrySnow
[fix](Nereids) remove getTableInMinidumpCache temporary
684e7d4
bug introduced by apache#18747
getTableInMinidumpCache use wrong way to compare table's qualified name.
we remove it temporary since it not use in productive env anymore
@morrySnowmorrySnow mentioned this pull request May 29, 2024
Merged
morrySnow added a commit to morrySnow/incubator-doris that referenced this pull request May 29, 2024
@morrySnow
[fix](Nereids) remove getTableInMinidumpCache temporary
11e139f
pick from master apache#35571
bug introduced by apache#18747
getTableInMinidumpCache use wrong way to compare table's qualified name.
we remove it temporary since it not use in productive env anymore
@morrySnowmorrySnow mentioned this pull request May 29, 2024
Merged
morrySnow added a commit to morrySnow/incubator-doris that referenced this pull request May 29, 2024
@morrySnow
[fix](Nereids) remove getTableInMinidumpCache temporary
d24cd9e
bug introduced by apache#18747
getTableInMinidumpCache use wrong way to compare table's qualified name.
we remove it temporary since it not use in productive env anymore
morrySnow added a commit that referenced this pull request May 29, 2024
@morrySnow
[fix](Nereids) remove getTableInMinidumpCache temporary (#35578)
0443fc1
pick from master #35571
bug introduced by #18747
getTableInMinidumpCache use wrong way to compare table's qualified name.
we remove it temporary since it not use in productive env anymore
morrySnow added a commit that referenced this pull request May 29, 2024
@morrySnow
[fix](Nereids) remove getTableInMinidumpCache temporary (#35571)
f6e624a
bug introduced by #18747
getTableInMinidumpCache use wrong way to compare table's qualified name.
we remove it temporary since it not use in productive env anymore
yiguolei pushed a commit that referenced this pull request May 29, 2024
@morrySnow@Doris-Extras
[fix](Nereids) remove getTableInMinidumpCache temporary (#35571)
f6e8c53
bug introduced by #18747
getTableInMinidumpCache use wrong way to compare table's qualified name.
we remove it temporary since it not use in productive env anymore
dataroaring pushed a commit that referenced this pull request May 31, 2024
@morrySnow@dataroaring
[fix](Nereids) remove getTableInMinidumpCache temporary (#35571)
6aff5b1
bug introduced by #18747
getTableInMinidumpCache use wrong way to compare table's qualified name.
we remove it temporary since it not use in productive env anymore
mongo360 pushed a commit to mongo360/doris that referenced this pull request Aug 16, 2024
@morrySnowweixingyu12
[fix](Nereids) remove getTableInMinidumpCache temporary (apache#35578)
219a149
pick from master apache#35571
bug introduced by apache#18747
getTableInMinidumpCache use wrong way to compare table's qualified name.
we remove it temporary since it not use in productive env anymore
HappenLee pushed a commit to HappenLee/incubator-doris that referenced this pull request Apr 24, 2026
@morrySnow@HappenLee
[fix](Nereids) remove getTableInMinidumpCache temporary (apache#35571)
c43cc8f
bug introduced by apache#18747
getTableInMinidumpCache use wrong way to compare table's qualified name.
we remove it temporary since it not use in productive env anymore
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@jackwenerjackwenerjackwener approved these changes

Assignees

No one assigned

Labels

area/nereidsreviewed

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@LiBinfeng-01@morrySnow@hello-stephen@jackwener