Skip to content

[Enhancement](point query optimize) improve performace of point query on primary keys - #15491

Merged
dataroaring merged 12 commits into
apache:masterfrom
eldenmoon:pq-opt
Jan 20, 2023
Merged

[Enhancement](point query optimize) improve performace of point query on primary keys #15491
dataroaring merged 12 commits into
apache:masterfrom
eldenmoon:pq-opt

Conversation

@eldenmoon

@eldenmooneldenmoon commented Dec 29, 2022•
edited
Loading

Copy link
Copy Markdown
Member

Proposed changes

Issue Number: close#15571

Problem summary

Describe your changes.

Checklist(Required)

  1. Does it affect the original behavior:
    • Yes
    • No
    • I don't know
  2. Has unit tests been added:
    • Yes
    • No
    • No Need
  3. Has document been added or modified:
    • Yes
    • No
    • No Need
  4. Does it need to update dependencies:
    • Yes
    • No
  5. Are there any changes that cannot be rolled back:
    • Yes (If Yes, please explain WHY)
    • No

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

@github-actionsgithub-actionsBot added area/planner Issues or PRs related to the query planner area/vectorization kind/test labels Dec 29, 2022
@eldenmoon
eldenmoonforce-pushed the pq-opt branch 2 times, most recently from 960efac to 066475fCompareDecember 29, 2022 13:51
@kpfly

kpfly commented Dec 29, 2022

Copy link
Copy Markdown

Add documents later pls

github-actions[bot]
github-actionsBot reviewed Dec 29, 2022
View reviewed changes

@github-actionsgithub-actionsBot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

clang-tidy made some suggestions

Comment threadbe/src/olap/tablet_schema.cpp
Comment threadbe/src/service/tablet_lookup_metric.h Outdated
Comment threadbe/src/util/jsonb_document.h
Comment threadbe/src/util/jsonb_document.h
Comment threadbe/src/util/jsonb_document.h
Comment threadbe/test/vec/jsonb/serialize_test.cpp Outdated
Comment threadbe/test/vec/jsonb/serialize_test.cpp
Comment threadbe/test/vec/jsonb/serialize_test.cpp Outdated
Comment threadbe/test/vec/jsonb/serialize_test.cpp Outdated
Comment threadbe/test/vec/jsonb/serialize_test.cpp Outdated
github-actions[bot]
github-actionsBot reviewed Jan 3, 2023
View reviewed changes

@github-actionsgithub-actionsBot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

clang-tidy made some suggestions

Comment threadbe/src/util/jsonb_document.h Outdated
Comment threadbe/src/util/jsonb_document.h Outdated
Comment threadbe/src/util/jsonb_document.h Outdated
Comment threadbe/src/util/jsonb_document.h Outdated
Comment threadbe/src/util/jsonb_document.h Outdated
Comment threadbe/test/vec/jsonb/serialize_test.cpp
Comment threadbe/test/vec/jsonb/serialize_test.cpp
Comment threadbe/test/vec/jsonb/serialize_test.cpp
Comment threadbe/test/vec/jsonb/serialize_test.cpp
Comment threadbe/test/vec/jsonb/serialize_test.cpp
@github-actionsgithub-actionsBot added the kind/docs Categorizes issue or PR as related to documentation. label Jan 3, 2023
@eldenmooneldenmoon changed the title Pq opt[Enhancement] improve performace of point query on primary keys Jan 3, 2023
@eldenmoon
eldenmoon marked this pull request as ready for review January 3, 2023 10:30
@eldenmooneldenmoon changed the title [Enhancement] improve performace of point query on primary keys [Enhancement](point query optimize) improve performace of point query on primary keys Jan 3, 2023
@hello-stephen

hello-stephen commented Jan 3, 2023•
edited
Loading

Copy link
Copy Markdown
Contributor

TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 33.71 seconds
load time: 492 seconds
storage size: 17121294106 Bytes
https://doris-community-test-1308700295.cos.ap-hongkong.myqcloud.com/tmp/20230119091247_clickbench_pr_84005.html

xiaokang
xiaokang reviewed Jan 5, 2023
View reviewed changes
Comment threadbe/src/util/jsonb_stream.h
xiaokang
xiaokang reviewed Jan 5, 2023
View reviewed changes
Comment threadbe/src/util/jsonb_writer.h Outdated
xiaokang
xiaokang reviewed Jan 9, 2023
View reviewed changes

@xiaokangxiaokang left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

be code reviewed.

wonderful job for point query performance.

Comment threadbe/src/util/mysql_global.h Outdated
Comment threadbe/src/util/mysql_global.h Outdated
Comment threadbe/src/vec/jsonb/serialize.cpp
Comment threadbe/src/vec/jsonb/serialize.cpp
Comment threadbe/src/vec/jsonb/serialize.cpp Outdated
Comment threadbe/src/runtime/descriptors.h
Comment threadbe/src/runtime/datetime_value.h
Comment threadbe/src/olap/delta_writer.cpp Outdated
Comment threadbe/src/olap/rowset/segment_v2/segment_iterator.cpp Outdated
Comment threadbe/src/util/mysql_row_buffer.cpp Outdated
@eldenmoon
eldenmoonforce-pushed the pq-opt branch 2 times, most recently from 494aa9c to cb2c970CompareJanuary 10, 2023 02:56
eldenmoon
eldenmoon commented Jan 10, 2023
View reviewed changes
Comment threadfe/fe-core/src/main/java/org/apache/doris/qe/PointQueryExec.java
Comment threadbe/src/util/jsonb_stream.h
Comment threadbe/src/vec/jsonb/serialize.cpp
Comment threadbe/src/vec/jsonb/serialize.cpp
Comment threadbe/src/util/mysql_global.h Outdated
Comment threadbe/src/runtime/descriptors.h
Comment threadbe/src/runtime/datetime_value.h
Comment threadbe/src/olap/delta_writer.cpp Outdated
Comment threadbe/src/olap/rowset/segment_v2/segment_iterator.cpp Outdated
Comment threadbe/src/util/mysql_row_buffer.cpp Outdated
github-actions[bot]
github-actionsBot reviewed Jan 10, 2023
View reviewed changes

@github-actionsgithub-actionsBot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

clang-tidy made some suggestions

Comment threadbe/src/runtime/descriptors.h
Comment threadbe/src/runtime/descriptors.h
Comment threadbe/src/util/jsonb_writer.h
Comment threadbe/test/vec/jsonb/serialize_test.cpp
Comment threadbe/test/vec/jsonb/serialize_test.cpp
Comment threadbe/test/vec/jsonb/serialize_test.cpp
Comment threadbe/test/vec/jsonb/serialize_test.cpp
Comment threadbe/test/vec/jsonb/serialize_test.cpp
@eldenmoon

eldenmoon commented Jan 16, 2023

Copy link
Copy Markdown
MemberAuthor

@xiaokang PR modified according to code review, lets' continue ?

xiaokang
xiaokang reviewed Jan 16, 2023
View reviewed changes

@xiaokangxiaokang left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

fe code reviewed

Comment threadfe/fe-core/src/main/cup/sql_parser.cup
Comment threadgensrc/proto/internal_service.proto Outdated
Comment threadgensrc/proto/internal_service.proto
Comment threadgensrc/proto/internal_service.proto
Comment threadfe/fe-core/src/main/java/org/apache/doris/qe/StmtExecutor.java
Comment threadfe/fe-core/src/main/java/org/apache/doris/planner/OriginalPlanner.java
Comment threadfe/fe-core/src/main/java/org/apache/doris/catalog/Env.java
Comment threadfe/fe-core/src/main/java/org/apache/doris/analysis/PlaceHolderExpr.java
Comment threadfe/fe-core/src/main/java/org/apache/doris/analysis/LiteralExpr.java
Comment threadfe/fe-core/src/main/java/org/apache/doris/analysis/DateLiteral.java
@eldenmoon
eldenmoonforce-pushed the pq-opt branch 5 times, most recently from dffab70 to 2b22b71CompareJanuary 18, 2023 11:56
xiaokang
xiaokang approved these changes Jan 19, 2023
View reviewed changes

@xiaokangxiaokang left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions

github-actionsBot commented Jan 19, 2023

Copy link
Copy Markdown
Contributor

PR approved by anyone and no changes requested.

dataroaring
dataroaring approved these changes Jan 20, 2023
View reviewed changes

@dataroaringdataroaring left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. We should complement docs later and we'd better let rowset and segment do not handle ifs for rowstore.

Because it is a independent with others, and column update of mow depends on it, so I merge it and notify the author that he should adjust some code.

@github-actionsgithub-actionsBot added the approved Indicates a PR has been approved by one committer. label Jan 20, 2023
@dataroaring
dataroaring merged commit 116e174 into apache:masterJan 20, 2023
@github-actions

github-actionsBot commented Jan 20, 2023

Copy link
Copy Markdown
Contributor

PR approved by at least one committer and no changes requested.

dutyu pushed a commit to dutyu/doris that referenced this pull request Feb 1, 2023
@eldenmoonwangxiangyu@360shuke.com
[Enhancement](point query optimize) improve performace of point query…
8a8c894
… on primary keys (apache#15491)
1. support row format using codec of jsonb
2. short path optimize for point query
3. support prepared statement for point query
4. support mysql binary format
@bobhan1bobhan1 mentioned this pull request Nov 13, 2023
Merged
zhannngchen pushed a commit that referenced this pull request Nov 13, 2023
@bobhan1
[fix](partial update) Fix NPE when the query statement of an update s…
2853efd
…tatement is a point query in `OriginPlanner` (#26881)
close#26882
We should not use the singleNodePlan to generate the rootPlanFragment if the query is inside a insert statement or distributedPlanner will be null.
introduced in #15491
seawinde pushed a commit to seawinde/doris that referenced this pull request Nov 14, 2023
@bobhan1@seawinde
[fix](partial update) Fix NPE when the query statement of an update s…
dc63cba
…tatement is a point query in `OriginPlanner` (apache#26881)
closeapache#26882
We should not use the singleNodePlan to generate the rootPlanFragment if the query is inside a insert statement or distributedPlanner will be null.
introduced in apache#15491
seawinde pushed a commit to seawinde/doris that referenced this pull request Nov 14, 2023
@bobhan1@seawinde
[fix](partial update) Fix NPE when the query statement of an update s…
4a7ee3d
…tatement is a point query in `OriginPlanner` (apache#26881)
closeapache#26882
We should not use the singleNodePlan to generate the rootPlanFragment if the query is inside a insert statement or distributedPlanner will be null.
introduced in apache#15491
XuJianxu pushed a commit to XuJianxu/doris that referenced this pull request Dec 14, 2023
@bobhan1胥剑旭
[fix](partial update) Fix NPE when the query statement of an update s…
d725db5
…tatement is a point query in `OriginPlanner` (apache#26881)
closeapache#26882
We should not use the singleNodePlan to generate the rootPlanFragment if the query is inside a insert statement or distributedPlanner will be null.
introduced in apache#15491
@eldenmooneldenmoon mentioned this pull request Apr 29, 2024
Merged
@eldenmooneldenmoon mentioned this pull request Jun 4, 2024
Merged
eldenmoon added a commit that referenced this pull request Jun 13, 2024
@eldenmoon
[Feature](Point Query) fully support in nereids (#35823)
2f517a9
step2: support short circuit query and short circuit(full) prepared
step1 PR: #35318
original implementation in legacy planner: #15491
dataroaring pushed a commit that referenced this pull request Jun 13, 2024
@eldenmoon@dataroaring
[Feature](Point Query) fully support in nereids (#35823)
166ef6d
step2: support short circuit query and short circuit(full) prepared
step1 PR: #35318
original implementation in legacy planner: #15491
HappenLee pushed a commit to HappenLee/incubator-doris that referenced this pull request Apr 24, 2026
@eldenmoon@HappenLee
[Feature](Point Query) fully support in nereids (apache#35823)
489de70
step2: support short circuit query and short circuit(full) prepared
step1 PR: apache#35318
original implementation in legacy planner: apache#15491
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@github-actionsgithub-actions[bot]github-actions[bot] left review comments
@xiaokangxiaokangxiaokang approved these changes
@dataroaringdataroaringdataroaring approved these changes

Assignees

No one assigned

Labels

approvedIndicates a PR has been approved by one committer.area/plannerIssues or PRs related to the query plannerarea/vectorizationkind/docsCategorizes issue or PR as related to documentation.kind/testreviewed

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

[Enhancement] improve performace of point query on primary keys

5 participants

@eldenmoon@kpfly@hello-stephen@xiaokang@dataroaring