Skip to content

[FIX](func) fix count distinct do not support arr/map/struct - #25483

Merged
xiaokang merged 3 commits into
apache:masterfrom
amorynan:arr-map-struct-ban-count-disctinct
Oct 19, 2023
Merged

[FIX](func) fix count distinct do not support arr/map/struct#25483
xiaokang merged 3 commits into
apache:masterfrom
amorynan:arr-map-struct-ban-count-disctinct

Conversation

@amorynan

Copy link
Copy Markdown
Contributor

Proposed changes

Issue Number: close #xxx
with old optimizer:
we would met core when we sql: select count(distinct arr or map or struct) from table
like this:

It's highly recommended that you fix the library with 'execstack -c <libfile>', or link it with '-z noexecstack'.
F20231016 17:46:58.601244 3568578 column.h:331] get_max_row_byte_size not supported
*** Check failure stack trace: ***
@ 0x557c8826e206 google::LogMessage::SendToLog()
@ 0x557c8826a7d0 google::LogMessage::Flush()
@ 0x557c8826ea49 google::LogMessageFatal::~LogMessageFatal()
@ 0x557c5f1d88b6 doris::vectorized::IColumn::get_max_row_byte_size()
@ 0x557c6fa60270 doris::vectorized::ColumnNullable::get_max_row_byte_size()
@ 0x557c7984abc3 doris::vectorized::MethodSerialized<>::init_serialized_keys()
@ 0x557c70277e33 doris::vectorized::DistinctAggregationNode::_emplace_into_hash_table_to_distinct()::$_0::operator()<>()
@ 0x557c70277a6d std::__invoke_impl<>()
@ 0x557c70277a1d std::__invoke<>()
@ 0x557c7027716d (unknown)
@ 0x557c7027710a (unknown)
@ 0x557c70277073 (unknown)
@ 0x557c70276f83 doris::vectorized::DistinctAggregationNode::_emplace_into_hash_table_to_distinct()
@ 0x557c70275b52 doris::vectorized::DistinctAggregationNode::_distinct_pre_agg_with_serialized_key()
@ 0x557c877f0315 doris::pipeline::DistinctStreamingAggSinkOperator::sink()
@ 0x557c8809bb75 doris::pipeline::PipelineTask::execute()
@ 0x557c8823d3af doris::pipeline::TaskScheduler::_do_work()
@ 0x557c8824c0e9 std::__invoke_impl<>()
@ 0x557c8824bf55 std::__invoke<>()
@ 0x557c8824bec4 _ZNSt5_BindIFMN5doris8pipeline13TaskSchedulerEFvmEPS2_mEE6__callIvJEJLm0ELm1EEEET_OSt5tupleIJDpT0_EESt12_Index_tupleIJXspT1_EEE
@ 0x557c8824bd2e std::_Bind<>::operator()<>()
@ 0x557c8824bc45 std::__invoke_impl<>()
@ 0x557c8824bbe5 _ZSt10__invoke_rIvRSt5_BindIFMN5doris8pipeline13TaskSchedulerEFvmEPS3_mEEJEENSt9enable_ifIX16is_invocable_r_vIT_T0_DpT1_EESB_E4typeEOSC_DpOSD_
@ 0x557c8824b8dd std::_Function_handler<>::_M_invoke()
@ 0x557c5ead63e3 std::function<>::operator()()
@ 0x557c61f638e9 doris::FunctionRunnable::run()
@ 0x557c61f4f68e doris::ThreadPool::dispatch_thread()
@ 0x557c61f760e4 std::__invoke_impl<>()
@ 0x557c61f75fbd std::__invoke<>()
@ 0x557c61f75f45 _ZNSt5_BindIFMN5doris10ThreadPoolEFvvEPS1_EE6__callIvJEJLm0EEEET_OSt5tupleIJDpT0_EESt12_Index_tupleIJXspT1_EEE
@ 0x557c61f75dee std::_Bind<>::operator()<>()
@ 0x557c61f75d05 std::__invoke_impl<>()

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

@amorynanamorynan changed the title [FIX]update it[FIX](func) fix count distinct do not support arr/map/structOct 16, 2023
@amorynan

Copy link
Copy Markdown
ContributorAuthor

run buildall

@doris-robot

Copy link
Copy Markdown

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 44.78 seconds
stream load tsv: 551 seconds loaded 74807831229 Bytes, about 129 MB/s
stream load json: 21 seconds loaded 2358488459 Bytes, about 107 MB/s
stream load orc: 64 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 28.8 seconds inserted 10000000 Rows, about 347K ops/s
storage size: 17162154828 Bytes

xiaokang
xiaokang previously approved these changes Oct 17, 2023

@xiaokangxiaokang left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions

Copy link
Copy Markdown
Contributor

PR approved by at least one committer and no changes requested.

@github-actionsgithub-actionsBot added approved Indicates a PR has been approved by one committer. reviewed labels Oct 17, 2023
@github-actions

Copy link
Copy Markdown
Contributor

PR approved by anyone and no changes requested.

@github-actionsgithub-actionsBot removed the approved Indicates a PR has been approved by one committer. label Oct 19, 2023
@amorynan

Copy link
Copy Markdown
ContributorAuthor

run buildall

@amorynan

Copy link
Copy Markdown
ContributorAuthor

run feut

@doris-robot

Copy link
Copy Markdown

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 46.73 seconds
stream load tsv: 558 seconds loaded 74807831229 Bytes, about 127 MB/s
stream load json: 20 seconds loaded 2358488459 Bytes, about 112 MB/s
stream load orc: 65 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 33 seconds loaded 861443392 Bytes, about 24 MB/s
insert into select: 29.0 seconds inserted 10000000 Rows, about 344K ops/s
storage size: 17162295950 Bytes

@eldenmooneldenmoon left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actionsgithub-actionsBot added the approved Indicates a PR has been approved by one committer. label Oct 19, 2023
@github-actions

Copy link
Copy Markdown
Contributor

PR approved by at least one committer and no changes requested.

@xiaokangxiaokang left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@xiaokangxiaokang added the usercase Important user case type label label Oct 19, 2023
@xiaokang
xiaokang merged commit ed713c2 into apache:masterOct 19, 2023
dutyu pushed a commit to dutyu/doris that referenced this pull request Oct 28, 2023
@xiaokangxiaokang mentioned this pull request Dec 4, 2023
XuJianxu pushed a commit to XuJianxu/doris that referenced this pull request Dec 14, 2023
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

approvedIndicates a PR has been approved by one committer.dev/2.0.3-mergedp0_creviewedusercaseImportant user case type label

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants

@amorynan@doris-robot@xiaokang@eldenmoon