Uh oh!
There was an error while loading. Please reload this page.
[Vectorized][Feature] add ColumnHLL to support hll type - #7828
Merged
Conversation
| namespace doris::vectorized { | ||
| size_t DataTypeHLL::serialize(const IColumn& column, PColumn* pcolumn) const { |
Contributor
There was a problem hiding this comment.
The serialize() method has been refactored in #7939 , please change it after that PR being merged.
| struct is_complex<DataTypeBitMap::FieldType> : std::true_type {}; | ||
| template <> | ||
| struct is_complex<DataTypeHLL::FieldType> : std::true_type {}; |
Contributor
There was a problem hiding this comment.
Why define HLL in data_type_bitmap?
ContributorAuthor
There was a problem hiding this comment.
Why define HLL in
data_type_bitmap?
I will deal with it
HappenLee
left a comment
Contributor
There was a problem hiding this comment.
some problem need to fix
| HyperLogLog& operator=(HyperLogLog&& other) { | ||
| if (this != &other) { | ||
| if(_registers) { |
| private: | ||
| DISALLOW_COPY_AND_ASSIGN(HyperLogLog); | ||
| //DISALLOW_COPY_AND_ASSIGN(HyperLogLog); |
| auto result = this->data(place).get(); | ||
| column.insert_data(result.c_str(), result.length()); | ||
| auto& column = static_cast<ColumnHLL&>(to); | ||
| column.get_data().push_back(this->data(place).get()); |
Contributor
There was a problem hiding this comment.
emplace_back to reduce unless copy
| static HyperLogLog init_value() { | ||
| auto hll = HyperLogLog::empty(); | ||
| return {hll.c_str(), hll.size()}; | ||
| return HyperLogLog(hll); |
| if (nullmap[i]) { | ||
| hash_string = HyperLogLog::empty(); | ||
| std::string hash_string = HyperLogLog::empty(); | ||
| res_data[i] = HyperLogLog(Slice(hash_string)); |
zhangstar333force-pushed
the
vec_hll2
branch
2 times, most recently
from
February 10, 2022 03:47
b8b3023 to
7c780c5Compare| string_slot->len = size; | ||
| } else if (slot_desc->type() == TYPE_HLL) { | ||
| auto hll_value = (HyperLogLog*)(data_ref.data); | ||
| LOG(INFO)<<"block hll string: "<<hll_value->to_string(); |
| auto string_slot = dst->get_string_slot(slot_desc->tuple_offset()); | ||
| string_slot->ptr = reinterpret_cast<char*>(pool->allocate(size)); | ||
| hll_value->serialize((uint8_t*)string_slot->ptr); | ||
| string_slot->len = size; |
Contributor
There was a problem hiding this comment.
set string_slot->len to serialize size
| for (size_t i = 0; i < column_num; ++i) { | ||
| auto& hll = const_cast<HyperLogLog&>(data_column.get_element(i)); | ||
| hll.serialize(reinterpret_cast<uint8_t*>(buf)); |
Contributor
There was a problem hiding this comment.
the size is not match to max serialized size may cause error
| auto& hll = const_cast<HyperLogLog&>(data_column.get_element(i)); | ||
| allocate_content_size += hll.max_serialized_size(); | ||
| std::string result(hll.max_serialized_size(), '0'); | ||
| size_t actual_size = hll.serialize((uint8_t*)result.c_str()); |
Contributor
There was a problem hiding this comment.
should not serialize two times
Contributor
PR approved by at least one committer and no changes requested. |
Contributor
PR approved by anyone and no changes requested. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Proposed changes
Issue Number: close#7826
Problem Summary:
Describe the overview of changes.
Checklist(Required)
Further comments
If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...