[FIX](complex-type) fix agg table with complex type with replace state by amorynan · Pull Request #24873 · apache/doris

amorynan · 2023-09-25T10:36:48Z

Proposed changes

Issue Number: close #xxx
when we use replace agg state which we only supported in complex type, fix be not implement replace_column_data with agg value to iterator . so we will got a be core below this

F20230924 10:40:22.801700 1080720 column_array.h:234] replace_column_data_default not implemented
*** Check failure stack trace: ***
@ 0x564542ab01e6 google::LogMessage::SendToLog()
@ 0x564542aac7b0 google::LogMessage::Flush()
@ 0x564542ab0a29 google::LogMessageFatal::~LogMessageFatal()
@ 0x56452f2d911a doris::vectorized::ColumnArray::replace_column_data_default()
@ 0x56452f41fc76 doris::vectorized::ColumnNullable::replace_column_data()
@ 0x5645416c26b3 doris::vectorized::VerticalBlockReader::_copy_agg_data()
@ 0x5645416c1725 doris::vectorized::VerticalBlockReader::_update_agg_data()
@ 0x5645416c15f6 doris::vectorized::VerticalBlockReader::_append_agg_data()
@ 0x5645416c0b27 doris::vectorized::VerticalBlockReader::_agg_key_next_block()
@ 0x564520b3dce7 doris::vectorized::VerticalBlockReader::next_block_with_aggregation()
@ 0x564520b2d28b doris::Merger::vertical_compact_one_group()
@ 0x564520b3054a doris::Merger::vertical_merge_rowsets()
@ 0x564520b4e5e9 doris::Compaction::do_compaction_impl()
@ 0x564520b4c03f doris::Compaction::do_compaction()
@ 0x564520ce7947 doris::CumulativeCompaction::execute_compact_impl()
@ 0x564520b4b77b doris::Compaction::execute_compact()
@ 0x564520de8bd5 doris::Tablet::execute_compaction()
@ 0x564520851d76 doris::StorageEngine::_submit_compaction_task()::$_0::operator()()
@ 0x564520851bf5 std::__invoke_impl<>()
@ 0x564520851b95 _ZSt10__invoke_rIvRZN5doris13StorageEngine23_submit_compaction_taskESt10shared_ptrINS0_6TabletEENS0_14CompactionTypeEbE3$_0JEENSt9enable_ifIX16is_invocable_r_vIT_T0_DpT1_EES9_E4typeEOSA_DpOSB_
@ 0x5645208519ad std::_Function_handler<>::_M_invoke()
@ 0x56451ea937a3 std::function<>::operator()()
@ 0x564521e63279 doris::FunctionRunnable::run()
@ 0x564521e4f01e doris::ThreadPool::dispatch_thread()
@ 0x564521e759f4 std::__invoke_impl<>()
@ 0x564521e758cd std::__invoke<>()
@ 0x564521e75855 _ZNSt5_BindIFMN5doris10ThreadPoolEFvvEPS1_EE6__callIvJEJLm0EEEET_OSt5tupleIJDpT0_EESt12_Index_tupleIJXspT1_EEE
@ 0x564521e756fe std::_Bind<>::operator()<>()
@ 0x564521e75615 std::__invoke_impl<>()
@ 0x564521e755b5 _ZSt10__invoke_rIvRSt5_BindIFMN5doris10ThreadPoolEFvvEPS2_EEJEENSt9enable_ifIX16is_invocable_r_vIT_T0_DpT1_EESA_E4typeEOSB_DpOSC_
@ 0x564521e752dd std::_Function_handler<>::_M_invoke()
@ 0x56451ea937a3 std::function<>::operator()()

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

github-actions

clang-tidy made some suggestions

Sign in to view


- void replace_column_data(const IColumn&, size_t row, size_t self_row = 0) override {
- LOG(FATAL) << "replace_column_data not implemented";
+ void replace_column_data(const IColumn& rhs, size_t row, size_t self_row = 0) override {


github-actions · 2023-09-25T10:46:55Z


- void replace_column_data(const IColumn&, size_t row, size_t self_row = 0) override {
- LOG(FATAL) << "replace_column_data not implemented";
+ void replace_column_data(const IColumn& rhs, size_t row, size_t self_row = 0) override {


warning: method 'replace_column_data' can be made const [readability-make-member-function-const]
Suggested change
voidreplace_column_data(constIColumn&rhs, size_trow, size_tself_row=0) override {
voidreplace_column_data(constIColumn&rhs, size_trow, size_tself_row=0) constoverride {

github-actions · 2023-09-25T10:46:55Z

 }
 }

+ void replace_batch_column_data(const IColumn& rhs, size_t num_rows, size_t row,


warning: method 'replace_batch_column_data' can be made const [readability-make-member-function-const]
be/src/vec/columns/column_nullable.h:360:
- size_t self_row = 0) override {+ size_t self_row = 0) const override {

github-actions · 2023-09-25T10:46:55Z

 }
- void replace_column_data(const IColumn&, size_t row, size_t self_row = 0) override {
- LOG(FATAL) << "replace_column_data not implemented";
+ void replace_column_data(const IColumn& rhs, size_t row, size_t self_row = 0) override {


warning: method 'replace_column_data' can be made const [readability-make-member-function-const]
Suggested change
voidreplace_column_data(constIColumn&rhs, size_trow, size_tself_row=0) override {
voidreplace_column_data(constIColumn&rhs, size_trow, size_tself_row=0) constoverride {

github-actions · 2023-09-25T10:46:55Z

+ }
 }
+
+ void replace_batch_column_data(const IColumn& rhs, size_t num_rows, size_t row,


warning: method 'replace_batch_column_data' can be made const [readability-make-member-function-const]
be/src/vec/columns/column_struct.h:143:
- size_t self_row = 0) override {+ size_t self_row = 0) const override {

github-actions · 2023-09-25T11:05:32Z

PR approved by at least one committer and no changes requested.

github-actions · 2023-09-25T11:05:34Z

PR approved by anyone and no changes requested.

amorynan · 2023-09-25T11:41:06Z

run buildall

doris-robot · 2023-09-25T12:41:26Z

TeamCity be ut coverage result:
Function Coverage: 36.29% (8111/22349)
Line Coverage: 28.42% (64870/228255)
Region Coverage: 27.31% (33641/123182)
Branch Coverage: 23.97% (17172/71648)
Coverage Report: http://coverage.selectdb-in.cc/coverage/a71b0bb270d0764b27954817c14cc21917a3932a_a71b0bb270d0764b27954817c14cc21917a3932a/report/index.html

doris-robot · 2023-09-25T13:05:38Z

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 45.3 seconds
stream load tsv: 554 seconds loaded 74807831229 Bytes, about 128 MB/s
stream load json: 20 seconds loaded 2358488459 Bytes, about 112 MB/s
stream load orc: 65 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 31 seconds loaded 861443392 Bytes, about 26 MB/s
insert into select: 28.7 seconds inserted 10000000 Rows, about 348K ops/s
storage size: 17162332101 Bytes

amorynan · 2023-09-28T08:22:09Z

run buildall

doris-robot · 2023-09-28T09:04:05Z

TeamCity be ut coverage result:
Function Coverage: 36.30% (8145/22436)
Line Coverage: 28.45% (65138/228986)
Region Coverage: 27.38% (33743/123243)
Branch Coverage: 24.04% (17213/71614)
Coverage Report: http://coverage.selectdb-in.cc/coverage/7a5c4452f52c7452b0f19e0b100616882ac5c233_7a5c4452f52c7452b0f19e0b100616882ac5c233/report/index.html

doris-robot · 2023-09-28T09:18:51Z

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 45.95 seconds
stream load tsv: 564 seconds loaded 74807831229 Bytes, about 126 MB/s
stream load json: 20 seconds loaded 2358488459 Bytes, about 112 MB/s
stream load orc: 64 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 33 seconds loaded 861443392 Bytes, about 24 MB/s
insert into select: 29.0 seconds inserted 10000000 Rows, about 344K ops/s
storage size: 17162344204 Bytes

HappenLee

LGTM

github-actions · 2023-09-28T10:29:08Z

PR approved by at least one committer and no changes requested.

amorynan · 2023-09-28T10:36:42Z

run p1

apache#24873) fix agg table with complex type with replace state

#24873) fix agg table with complex type with replace state

fix agg table with complex type
a71b0bb

github-actionsBot reviewed Sep 25, 2023
View reviewed changes

BiteTheDDDDt previously approved these changes Sep 25, 2023
View reviewed changes

github-actionsBot added the approved Indicates a PR has been approved by one committer. label Sep 25, 2023

github-actionsBot added the reviewed label Sep 25, 2023

fix array/map agg
7a5c445

amorynan dismissed BiteTheDDDDt’s stale review via 7a5c445September 28, 2023 01:18

github-actionsBot removed the approved Indicates a PR has been approved by one committer. label Sep 28, 2023

HappenLee approved these changes Sep 28, 2023
View reviewed changes

github-actionsBot added the approved Indicates a PR has been approved by one committer. label Sep 28, 2023

BiteTheDDDDt approved these changes Oct 3, 2023
View reviewed changes

BiteTheDDDDt merged commit 10f0c63 into apache:masterOct 3, 2023

vinlee19 pushed a commit to vinlee19/doris that referenced this pull request Oct 7, 2023
[FIX](complex-type) fix agg table with complex type with replace state (
8e8af9b
apache#24873) fix agg table with complex type with replace state

xiaokang added dev/2.0.3 dev/2.0.3-merged and removed dev/2.0.3 labels Oct 7, 2023

xiaokang pushed a commit that referenced this pull request Oct 7, 2023
[FIX](complex-type) fix agg table with complex type with replace state (
5e2ecad
#24873) fix agg table with complex type with replace state

amorynan mentioned this pull request Apr 1, 2024
[FIX](agg) fix vertical_compaction_reader for agg table with array/map type #33130
Merged

	voidreplace_column_data(constIColumn&rhs, size_trow, size_tself_row=0) override {
	voidreplace_column_data(constIColumn&rhs, size_trow, size_tself_row=0) constoverride {

Uh oh!

Conversation

amorynan commented Sep 25, 2023

Proposed changes

Further comments

Uh oh!

github-actionsBot left a comment

Choose a reason for hiding this comment

Uh oh!

This comment was marked as outdated.

Uh oh!

github-actionsBotSep 25, 2023

Choose a reason for hiding this comment

Uh oh!

github-actionsBotSep 25, 2023

Choose a reason for hiding this comment

Uh oh!

github-actionsBotSep 25, 2023

Choose a reason for hiding this comment

Uh oh!

github-actionsBotSep 25, 2023

Choose a reason for hiding this comment

Uh oh!

github-actionsBot commented Sep 25, 2023

Uh oh!

github-actionsBot commented Sep 25, 2023

Uh oh!

amorynan commented Sep 25, 2023

Uh oh!

doris-robot commented Sep 25, 2023

Uh oh!

doris-robot commented Sep 25, 2023

Uh oh!

amorynan commented Sep 28, 2023

Uh oh!

doris-robot commented Sep 28, 2023

Uh oh!

doris-robot commented Sep 28, 2023

Uh oh!

HappenLee left a comment

Choose a reason for hiding this comment

Uh oh!

github-actionsBot commented Sep 28, 2023

Uh oh!

amorynan commented Sep 28, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants