Skip to content

[Feature] Support InPredicate in delete statement - #4006

Merged
morningman merged 22 commits into
apache:masterfrom
caiconghui:delete_in
Aug 6, 2020
Merged

[Feature] Support InPredicate in delete statement#4006
morningman merged 22 commits into
apache:masterfrom
caiconghui:delete_in

Conversation

@caiconghui

@caiconghuicaiconghui commented Jul 2, 2020

Copy link
Copy Markdown
Contributor

This PR is to add inPredicate support to delete statement, and add max_allowed_in_element_num_of_delete variable to limit element num of InPredicate in delete statement.

@caiconghui

caiconghui commented Jul 2, 2020

Copy link
Copy Markdown
ContributorAuthor

for #4008

@morningman

morningman commented Jul 2, 2020

Copy link
Copy Markdown
Contributor

In fact, we are also trying to implement a batch delete function that can support the deletion of a large number of specified keys (including specifying multiple values in inPredicate)

In your current implementation, if there are too many values in InPredicate (for example, tens of thousands), then each column must be looped through tens of thousands of conditions during query, which may be inefficient.

Maybe you can discuss it with @yangzhg about it.

morningman
morningman reviewed Jul 5, 2020
View reviewed changes
Comment threadfe/src/main/java/org/apache/doris/load/DeleteHandler.java
Comment threadfe/src/main/java/org/apache/doris/qe/SessionVariable.java Outdated
Comment threadbe/src/olap/delete_handler.cpp Outdated
Comment threadbe/src/olap/olap_cond.cpp Outdated
Comment threadbe/src/olap/olap_cond.cpp Outdated
Comment threadbe/src/olap/olap_cond.cpp Outdated
Comment threadbe/src/olap/olap_cond.h Outdated
wutiangan
wutiangan reviewed Jul 7, 2020
View reviewed changes
Comment threadfe/src/main/java/org/apache/doris/load/DeleteHandler.java Outdated
wutiangan
wutiangan reviewed Jul 7, 2020
View reviewed changes
Comment threadfe/src/main/java/org/apache/doris/load/DeleteHandler.java
wutiangan
wutiangan reviewed Jul 7, 2020
View reviewed changes
Comment threadfe/src/main/java/org/apache/doris/load/DeleteHandler.java
morningman
morningman reviewed Jul 29, 2020
View reviewed changes
Comment threadfe/fe-core/src/main/java/org/apache/doris/analysis/DeleteStmt.java Outdated
Comment threadfe/fe-core/src/main/java/org/apache/doris/analysis/DeleteStmt.java Outdated
Comment threadfe/fe-core/src/main/java/org/apache/doris/load/DeleteHandler.java Outdated
Comment threadbe/src/olap/olap_cond.cpp
wrapperField.release_field();
return ret;
}
case OP_NOT_IN: {

@morningmanmorningmanJul 29, 2020

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These 2 case (OP_IN and OP_NOT_IN) looks same?

@caiconghuicaiconghuiJul 29, 2020•
edited
Loading

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

one is "!="(line 196), the other is "=="(line 202)

Comment threadbe/src/olap/olap_cond.cpp Outdated
tcond.column_name.c_str(), operand.c_str(), op);
return res;
}
if (min_value_field == nullptr || f->cmp(min_value_field) > 0) {

@morningmanmorningmanJul 29, 2020

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am a little confused.

A->cmp(B) > 0 means A > B or A < B?

@caiconghuicaiconghuiJul 29, 2020

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

my bad, I will fix it A->cmp(B) > 0 means A > B

Comment threadbe/src/olap/olap_cond.h
// valid when op is OP_IN or OP_NOT_IN
typedef std::unordered_set<const WrapperField*, FieldHash, FieldEqual> FieldSet;
FieldSet operand_set;
WrapperField* min_value_field;

@morningmanmorningmanJul 29, 2020

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Add comment to explains these 2 new fields.

@caiconghuicaiconghuiJul 29, 2020

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done

@morningmanmorningman self-assigned this Jul 29, 2020
@morningmanmorningman added area/delete Issues or PRs related to DELETE operation kind/feature Categorizes issue or PR as related to a new feature. labels Jul 29, 2020
morningman
morningman previously approved these changes Aug 1, 2020
View reviewed changes

@morningmanmorningman left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@morningman

morningman commented Aug 1, 2020

Copy link
Copy Markdown
Contributor

UT failed.
FE：
DeleteStmtTest.testAnalyze

morningman
morningman reviewed Aug 1, 2020
View reviewed changes
Comment threadfe/fe-core/src/main/java/org/apache/doris/common/Config.java
Comment threadbe/src/olap/delete_handler.cpp
string condition_str = construct_sub_predicates(condition);
del_pred->add_sub_predicates(condition_str);
LOG(INFO) << "store one sub-delete condition. condition=" << condition_str;
if (condition.condition_values.size() > 1) {

@morningmanmorningmanAug 1, 2020

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it's better to check the condition.condition_op to see if this condition is InPredicate, not by the size of condition.condition_values.

@caiconghuicaiconghuiAug 1, 2020

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

because inPredicate with one element to stored as BinaryPredicate which is more more light and efficient.

morningman
morningman previously approved these changes Aug 1, 2020
View reviewed changes

@morningmanmorningman left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@morningmanmorningman added approved Indicates a PR has been approved by one committer. branch-0.13 PR which need to merge to branch 0.13 labels Aug 1, 2020
kangkaisen
kangkaisen reviewed Aug 3, 2020
View reviewed changes
Comment threadgensrc/proto/olap_file.proto
kangkaisen
kangkaisen previously approved these changes Aug 3, 2020
View reviewed changes

@kangkaisenkangkaisen left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

protobuf LGTM

morningman
morningman approved these changes Aug 4, 2020
View reviewed changes

@morningmanmorningman left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@morningman
morningman merged commit eefad13 into apache:masterAug 6, 2020
@EmmyMiao87EmmyMiao87 mentioned this pull request Aug 17, 2020
Closed
csun5285 added a commit to csun5285/doris that referenced this pull request Sep 23, 2025
@csun5285
[fix](inverted index) fix query error (apache#50860) (apache#4006)
b7b90d5
‌When the IN predicate in an expression contains parameters generated
through CAST operations, utilizing inverted indexes may produce
erroneous query results.
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@morningmanmorningmanmorningman approved these changes
@kangkaisenkangkaisenkangkaisen left review comments
+1 more reviewer
@wutianganwutianganwutiangan left review comments
Reviewers whose approvals may not affect merge requirements

Assignees

@morningmanmorningman

Labels

approvedIndicates a PR has been approved by one committer.area/deleteIssues or PRs related to DELETE operationbranch-0.13PR which need to merge to branch 0.13kind/featureCategorizes issue or PR as related to a new feature.

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@caiconghui@morningman@wutiangan@kangkaisen