Uh oh!
There was an error while loading. Please reload this page.
[Improve](expr)first support array_contains for expr push down inverted index - #32620
Conversation
doris-robot
commented
Mar 21, 2024
Thank you for your contribution to Apache Doris. Since 2024-03-18, the Document has been moved to doris-website. |
| return true; | ||
| } | ||
| bool SegmentIterator::_check_apply_by_inverted_index(ColumnId col_id) { |
This comment was marked as abuse.
This comment was marked as abuse.
Sorry, something went wrong.
Uh oh!
There was an error while loading. Please reload this page.
| const std::string& expr_name() const override { return _expr_name; } | ||
| bool is_all_ones(const roaring::Roaring& r) { |
There was a problem hiding this comment.
warning: method 'is_all_ones' can be made static [readability-convert-member-functions-to-static]
| boolis_all_ones(constroaring::Roaring&r) { | |
| staticboolis_all_ones(constroaring::Roaring&r) { |
| // 2. when meet 'and' conjunct, function with column b can not apply inverted index | ||
| // eg. a and hash(b)=1, if b can apply index, but hash(b)=1 is not for index, so b should not be extracted | ||
| // but a and array_contains(b, 1), b can be applied inverted index, which b can be extracted | ||
| Status eval_inverted_index( |
There was a problem hiding this comment.
warning: method 'eval_inverted_index' can be made static [readability-convert-member-functions-to-static]
| Statuseval_inverted_index( | |
| staticStatuseval_inverted_index( |
be/src/vec/exprs/vcompound_pred.h:74:
- uint32_t num_rows, roaring::Roaring* bitmap) const override {+ uint32_t num_rows, roaring::Roaring* bitmap) override {| std::vector<size_t>& args) override; | ||
| Status eval_inverted_index( | ||
| VExprContext* context, | ||
| const std::unordered_map<ColumnId, std::pair<vectorized::NameAndTypePair, |
There was a problem hiding this comment.
warning: parameter 2 is const-qualified in the function declaration; const-qualification of parameters only has an effect in function definitions [readability-avoid-const-params-in-decls]
| conststd::unordered_map<ColumnId, std::pair<vectorized::NameAndTypePair, | |
| std::unordered_map<ColumnId, std::pair<vectorized::NameAndTypePair, |
| * @return status not ok means execute failed. | ||
| */ | ||
| [[nodiscard]] Status eval_inverted_indexs( | ||
| const std::unordered_map<ColumnId, std::pair<vectorized::NameAndTypePair, |
There was a problem hiding this comment.
warning: parameter 1 is const-qualified in the function declaration; const-qualification of parameters only has an effect in function definitions [readability-avoid-const-params-in-decls]
| conststd::unordered_map<ColumnId, std::pair<vectorized::NameAndTypePair, | |
| std::unordered_map<ColumnId, std::pair<vectorized::NameAndTypePair, |
| return Status::OK(); | ||
| } | ||
| Status eval_inverted_index(FunctionContext* context, |
There was a problem hiding this comment.
warning: method 'eval_inverted_index' can be made static [readability-convert-member-functions-to-static]
| Statuseval_inverted_index(FunctionContext*context, | |
| staticStatuseval_inverted_index(FunctionContext*context, |
be/src/vec/functions/function.h:411:
- roaring::Roaring* bitmap) const override {+ roaring::Roaring* bitmap) override {Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
| // 2. when meet 'and' conjunct, function with column b can not apply inverted index | ||
| // eg. a and hash(b)=1, if b can apply index, but hash(b)=1 is not for index, so b should not be extracted | ||
| // but a and array_contains(b, 1), b can be applied inverted index, which b can be extracted | ||
| Status eval_inverted_index( |
There was a problem hiding this comment.
warning: method 'eval_inverted_index' can be made static [readability-convert-member-functions-to-static]
| Statuseval_inverted_index( | |
| staticStatuseval_inverted_index( |
be/src/vec/exprs/vcompound_pred.h:64:
- uint32_t num_rows, roaring::Roaring* bitmap) const override {+ uint32_t num_rows, roaring::Roaring* bitmap) override {| // specific language governing permissions and limitations | ||
| // under the License. | ||
| #include <gtest/gtest-message.h> |
There was a problem hiding this comment.
warning: 'gtest/gtest-message.h' file not found [clang-diagnostic-error]
#include<gtest/gtest-message.h>
^| return create_query_value<TYPE_STRING>(value, result_param); | ||
| case PrimitiveType::TYPE_IPV4: | ||
| return create_query_value<TYPE_IPV4>(value, result_param); | ||
| case PrimitiveType::TYPE_IPV6: |
There was a problem hiding this comment.
we do not support HLL/IPV6/IPV4 type for inverted index
| std::unordered_map<ColumnId, std::pair<vectorized::NameAndTypePair, InvertedIndexIterator*>> | ||
| iter_map; | ||
| for (auto col_id : _common_expr_columns) { |
There was a problem hiding this comment.
how about those non-array expr be processed in this situation?
There was a problem hiding this comment.
just will return NotSupport
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
amorynan
commented
Apr 18, 2024
run buildall |
amorynan
commented
Apr 18, 2024
run buildall |
1 similar comment
amorynan
commented
Apr 19, 2024
run buildall |
amorynan
commented
Apr 19, 2024
run buildall |
doris-robot
commented
Apr 19, 2024
TeamCity be ut coverage result: |
doris-robot
commented
Apr 19, 2024
TPC-H: Total hot run time: 37236 ms |
PR approved by at least one committer and no changes requested. |
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
amorynan
commented
Apr 28, 2024
run buildall |
PR approved by at least one committer and no changes requested. |
doris-robot
commented
Apr 28, 2024
TPC-DS: Total hot run time: 186909 ms |
doris-robot
commented
Apr 28, 2024
TeamCity be ut coverage result: |
amorynan
commented
Apr 28, 2024
run p0 |
amorynan
commented
Apr 28, 2024
run external |
Uh oh!
There was an error while loading. Please reload this page.
amorynan
commented
Apr 28, 2024
run buildall |
doris-robot
commented
Apr 28, 2024
TPC-H: Total hot run time: 41665 ms |
doris-robot
commented
Apr 28, 2024
TPC-DS: Total hot run time: 186120 ms |
doris-robot
commented
Apr 28, 2024
TeamCity be ut coverage result: |
PR approved by at least one committer and no changes requested. |
…ed index (apache#32620) in this pr we will do: support expr to eval inverted index inverted index reader param should be generic for expr
Proposed changes
in this pr we will do:
Further comments
If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...