Skip to content

[fix](Nereids): construct project with all slots in semi-semi-transpose-project rule - #17811

Merged
morrySnow merged 1 commit into
apache:masterfrom
keanji-x:fix_q20
Mar 16, 2023
Merged

[fix](Nereids): construct project with all slots in semi-semi-transpose-project rule#17811
morrySnow merged 1 commit into
apache:masterfrom
keanji-x:fix_q20

Conversation

@keanji-x

@keanji-xkeanji-x commented Mar 15, 2023

Copy link
Copy Markdown
Contributor

Proposed changes

error msg in tpch 20

SlotRef have invalid slot id: , desc: 22, slot_desc: tuple_desc_map: [Tuple(id=10 slots=[Slot(id=51 type=DECIMALV2(27, 9) col=-1, colname= null=(offset=0 mask=80)), Slot(id=52 type=INT col=-1, colname= null=(offset=0 mask=0)), Slot(id=53 type=INT col=-1, colname= null=(offset=0 mask=0)), Slot(id=54 type=INT col=-1, colname= null=(offset=0 mask=0)), Slot(id=55 type=INT col=-1, colname= null=(offset=0 mask=0))] has_varlen_slots=0)] tuple_id_map: [-1, -1, -1, -1, -1, -1, -1, -1, -1, -1, 0] tuple_is_nullable: [0] , desc_tbl: Slot(id=22 type=INT col=-1, colname= null=(offset=0 mask=0))

Before we only use slots in hashJoin conditions to construct projects, which may lost some slots in project, such as

LOGICAL_SEMI_JOIN_LOGICAL_JOIN_TRANSPOSE_PROJECT
LogicalJoin[1135] ( type=LEFT_SEMI_JOIN, markJoinSlotReference=Optional.empty, hashJoinConjuncts=[(PS_PARTKEY#0 = P_PARTKEY#6)], otherJoinConjuncts=[] )
|--LogicalProject[1128] ( distinct=false, projects=[PS_PARTKEY#0, PS_SUPPKEY#1], excepts=[], canEliminate=true )
| +--LogicalJoin[1120] ( type=LEFT_SEMI_JOIN, markJoinSlotReference=Optional.empty, hashJoinConjuncts=[(L_PARTKEY#17 = PS_PARTKEY#0), (L_SUPPKEY#18 = PS_SUPPKEY#1)], otherJoinConjuncts=[(cast(PS_AVAILQTY#2 as DECIMAL(27, 9)) > (0.5 * sum(L_QUANTITY))#33)] )
| |--GroupPlan( GroupId#2 )
| +--GroupPlan( GroupId#7 )
+--GroupPlan( GroupId#12 )
----------------------after----------------------
LogicalJoin[1141] ( type=LEFT_SEMI_JOIN, markJoinSlotReference=Optional.empty, hashJoinConjuncts=[(L_PARTKEY#17 = PS_PARTKEY#0), (L_SUPPKEY#18 = PS_SUPPKEY#1)], otherJoinConjuncts=[(cast(PS_AVAILQTY#2 as DECIMAL(27, 9)) > (0.5 * sum(L_QUANTITY))#33)] )
|--LogicalProject[1140] ( distinct=false, projects=[PS_PARTKEY#0, PS_SUPPKEY#1], excepts=[], canEliminate=true )
| +--LogicalJoin[1139] ( type=LEFT_SEMI_JOIN, markJoinSlotReference=Optional.empty, hashJoinConjuncts=[(PS_PARTKEY#0 = P_PARTKEY#6)], otherJoinConjuncts=[] )
| |--GroupPlan( GroupId#2 )
| +--GroupPlan( GroupId#12 )
+--GroupPlan( GroupId#7 )

PS_AVAILQTY#2 lost in project

Now we use all slots to construct projest

Problem summary

Describe your changes.

Checklist(Required)

  • Does it affect the original behavior
  • Has unit tests been added
  • Has document been added or modified
  • Does it need to update dependencies
  • Is this PR support rollback (If NO, please explain WHY)

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

Signed-off-by: xiejiann <jianxie0@gmail.com>
@keanji-x

Copy link
Copy Markdown
ContributorAuthor

run buildall

@github-actionsgithub-actionsBot added the approved Indicates a PR has been approved by one committer. label Mar 15, 2023
@github-actions

Copy link
Copy Markdown
Contributor

PR approved by at least one committer and no changes requested.

@github-actions

Copy link
Copy Markdown
Contributor

PR approved by anyone and no changes requested.

@hello-stephen

Copy link
Copy Markdown
Contributor

TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 34.04 seconds
stream load tsv: 467 seconds loaded 74807831229 Bytes, about 152 MB/s
stream load json: 41 seconds loaded 2358488459 Bytes, about 54 MB/s
stream load orc: 75 seconds loaded 1101869774 Bytes, about 14 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
https://doris-community-test-1308700295.cos.ap-hongkong.myqcloud.com/tmp/20230315083138_clickbench_pr_114517.html

@morrySnow
morrySnow merged commit c2edca7 into apache:masterMar 16, 2023
gnehil pushed a commit to gnehil/doris that referenced this pull request Apr 21, 2023
…e-project rule (apache#17811)
error msg in tpch 20
```
SlotRef have invalid slot id: , desc: 22, slot_desc: tuple_desc_map: [Tuple(id=10 slots=[Slot(id=51 type=DECIMALV2(27, 9) col=-1, colname= null=(offset=0 mask=80)), Slot(id=52 type=INT col=-1, colname= null=(offset=0 mask=0)), Slot(id=53 type=INT col=-1, colname= null=(offset=0 mask=0)), Slot(id=54 type=INT col=-1, colname= null=(offset=0 mask=0)), Slot(id=55 type=INT col=-1, colname= null=(offset=0 mask=0))] has_varlen_slots=0)] tuple_id_map: [-1, -1, -1, -1, -1, -1, -1, -1, -1, -1, 0] tuple_is_nullable: [0] , desc_tbl: Slot(id=22 type=INT col=-1, colname= null=(offset=0 mask=0))
```
Before we only use slots in `hashJoin` conditions to construct projects, which may lost some slots in `project`, such as ```
LOGICAL_SEMI_JOIN_LOGICAL_JOIN_TRANSPOSE_PROJECT
LogicalJoin[1135] ( type=LEFT_SEMI_JOIN, markJoinSlotReference=Optional.empty, hashJoinConjuncts=[(PS_PARTKEY#0 = P_PARTKEY#6)], otherJoinConjuncts=[] )
|--LogicalProject[1128] ( distinct=false, projects=[PS_PARTKEY#0, PS_SUPPKEY#1], excepts=[], canEliminate=true )
| +--LogicalJoin[1120] ( type=LEFT_SEMI_JOIN, markJoinSlotReference=Optional.empty, hashJoinConjuncts=[(L_PARTKEY#17 = PS_PARTKEY#0), (L_SUPPKEY#18 = PS_SUPPKEY#1)], otherJoinConjuncts=[(cast(PS_AVAILQTY#2 as DECIMAL(27, 9)) > (0.5 * sum(L_QUANTITY))apache#33)] )
| |--GroupPlan( GroupId#2 )
| +--GroupPlan( GroupId#7 )
+--GroupPlan( GroupId#12 )
----------------------after----------------------
LogicalJoin[1141] ( type=LEFT_SEMI_JOIN, markJoinSlotReference=Optional.empty, hashJoinConjuncts=[(L_PARTKEY#17 = PS_PARTKEY#0), (L_SUPPKEY#18 = PS_SUPPKEY#1)], otherJoinConjuncts=[(cast(PS_AVAILQTY#2 as DECIMAL(27, 9)) > (0.5 * sum(L_QUANTITY))apache#33)] )
|--LogicalProject[1140] ( distinct=false, projects=[PS_PARTKEY#0, PS_SUPPKEY#1], excepts=[], canEliminate=true )
| +--LogicalJoin[1139] ( type=LEFT_SEMI_JOIN, markJoinSlotReference=Optional.empty, hashJoinConjuncts=[(PS_PARTKEY#0 = P_PARTKEY#6)], otherJoinConjuncts=[] )
| |--GroupPlan( GroupId#2 )
| +--GroupPlan( GroupId#12 )
+--GroupPlan( GroupId#7 )
```
`PS_AVAILQTY#2` lost in project
Now we use all slots to construct projest
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

approvedIndicates a PR has been approved by one committer.area/nereidsreviewed

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants

@keanji-x@hello-stephen@jackwener@morrySnow