[fix](function) fix core when input not null array in foreach function by Mryange · Pull Request #37349 · apache/doris

Mryange · 2024-07-05T10:02:33Z

Proposed changes

error code

return creator_without_type::create<AggregateFunctionForEach>(transform_arguments, true,
nested_function);

"transform_arguments is an internal type of array. All internal types of the array are null, so an array that is not null was mistakenly treated as a null array."

core

INFO: java_cmd /mnt/disk2/yanxuecheng/java17/bin/java
INFO: jdk_version 17
start BE in local mode
F20240705 15:45:48.466758 3587508 assert_cast.h:45] Bad cast from type:doris::vectorized::ColumnArray* to doris::vectorized::ColumnNullable const*
*** Check failure stack trace: ***
@ 0x5612b1cf86a6 google::LogMessage::SendToLog()
@ 0x5612b1cf50f0 google::LogMessage::Flush()
@ 0x5612b1cf8ee9 google::LogMessageFatal::~LogMessageFatal()
@ 0x5612a3373e74 assert_cast<>()
@ 0x5612a778e60d doris::vectorized::AggregateFunctionNullUnaryInline<>::add_batch_single_place()
@ 0x5612b0d0dba1 doris::vectorized::AggFnEvaluator::execute_single_add()
@ 0x5612b0c62b52 doris::pipeline::AggSinkLocalState::_execute_without_key()
@ 0x5612b0d0a90c doris::pipeline::AggSinkLocalState::Executor<>::execute()
@ 0x5612b0c653df doris::pipeline::AggSinkOperatorX::sink()
@ 0x5612b1bf27c8 doris::pipeline::PipelineTask::execute()::$_1::operator()()
@ 0x5612b1bf15c6 doris::pipeline::PipelineTask::execute()
@ 0x5612b1c0e8e7 doris::pipeline::TaskScheduler::_do_work()
@ 0x5612b1c0f72c doris::pipeline::TaskScheduler::start()::$_0::operator()()
@ 0x5612b1c0f705 std::__invoke_impl<>()
@ 0x5612b1c0f6b5 _ZSt10__invoke_rIvRZN5doris8pipeline13TaskScheduler5startEvE3$_0JEENSt9enable_ifIX16is_invocable_r_vIT_T0_DpT1_EES6_E4typeEOS7_DpOS8_
@ 0x5612b1c0f5bd std::_Function_handler<>::_M_invoke()
@ 0x5612a1400905 std::function<>::operator()()
@ 0x5612a2ad6319 doris::FunctionRunnable::run()
@ 0x5612a2acf9b6 doris::ThreadPool::dispatch_thread()
@ 0x5612a2addf09 std::__invoke_impl<>()
@ 0x5612a2adde4d std::__invoke<>()
@ 0x5612a2adde1d _ZNSt5_BindIFMN5doris10ThreadPoolEFvvEPS1_EE6__callIvJEJLm0EEEET_OSt5tupleIJDpT0_EESt12_Index_tupleIJXspT1_EEE
@ 0x5612a2adddd6 std::_Bind<>::operator()<>()
@ 0x5612a2addda5 std::__invoke_impl<>()
@ 0x5612a2addd65 _ZSt10__invoke_rIvRSt5_BindIFMN5doris10ThreadPoolEFvvEPS2_EEJEENSt9enable_ifIX16is_invocable_r_vIT_T0_DpT1_EESA_E4typeEOSB_DpOSC_
@ 0x5612a2addbbd std::_Function_handler<>::_M_invoke()
@ 0x5612a1400905 std::function<>::operator()()
@ 0x5612a2abecff doris::Thread::supervise_thread()
@ 0x7f6d1c7841ca start_thread
@ 0x7f6d1d173e73 __GI___clone
@ (nil) (unknown)

doris-robot · 2024-07-05T10:02:39Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

Mryange · 2024-07-05T10:03:02Z

run buildall

github-actions · 2024-07-05T10:08:22Z

clang-tidy review says "All clean, LGTM! 👍"

doris-robot · 2024-07-05T10:35:27Z

TPC-H: Total hot run time: 40690 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 083604df5168f586dd6a5f37c7df475d9c03fba7, data reload: false
------ Round 1 ----------------------------------
q1	17633	4523	4310	4310
q2	2017	195	196	195
q3	10467	1191	1102	1102
q4	10186	737	833	737
q5	7469	2652	2699	2652
q6	221	138	136	136
q7	954	601	597	597
q8	9431	2061	2087	2061
q9	8801	6662	6631	6631
q10	9038	3751	3725	3725
q11	449	236	242	236
q12	491	225	223	223
q13	19313	3011	3018	3011
q14	264	231	236	231
q15	556	503	485	485
q16	511	398	375	375
q17	977	646	678	646
q18	8186	7603	7556	7556
q19	6137	1497	1343	1343
q20	653	334	314	314
q21	4971	3792	3900	3792
q22	382	340	332	332
Total cold run time: 119107 ms
Total hot run time: 40690 ms
----- Round 2, with runtime_filter_mode=off -----
q1	4440	4382	4374	4374
q2	374	298	284	284
q3	3290	3119	3083	3083
q4	2055	1765	1775	1765
q5	5709	5770	5629	5629
q6	242	137	134	134
q7	2444	1929	1834	1834
q8	3333	3508	3473	3473
q9	8780	8879	8761	8761
q10	4119	3839	3849	3839
q11	583	496	504	496
q12	826	629	616	616
q13	15986	3150	3202	3150
q14	321	303	259	259
q15	560	486	507	486
q16	524	454	420	420
q17	1835	1539	1522	1522
q18	8349	7966	7741	7741
q19	1784	1658	1547	1547
q20	3238	1908	1868	1868
q21	10945	4842	4816	4816
q22	638	558	563	558
Total cold run time: 80375 ms
Total hot run time: 56655 ms

doris-robot · 2024-07-05T10:46:43Z

TPC-DS: Total hot run time: 173309 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 083604df5168f586dd6a5f37c7df475d9c03fba7, data reload: false
query1	917	374	390	374
query2	6465	2544	2344	2344
query3	6636	207	219	207
query4	22833	17540	17314	17314
query5	3579	480	524	480
query6	258	169	166	166
query7	4591	309	293	293
query8	290	302	284	284
query9	8718	2467	2415	2415
query10	573	296	280	280
query11	10656	10076	10130	10076
query12	114	84	81	81
query13	1639	362	364	362
query14	10152	6854	7501	6854
query15	244	184	185	184
query16	7746	294	300	294
query17	1574	552	551	551
query18	1949	283	274	274
query19	182	151	157	151
query20	90	81	79	79
query21	213	132	125	125
query22	4410	4196	3997	3997
query23	33837	33844	33527	33527
query24	10732	2852	2867	2852
query25	573	394	379	379
query26	714	161	167	161
query27	2210	318	330	318
query28	5965	2169	2146	2146
query29	872	663	633	633
query30	258	157	159	157
query31	997	766	807	766
query32	93	52	53	52
query33	658	290	296	290
query34	892	503	500	500
query35	728	628	646	628
query36	1129	988	997	988
query37	140	80	85	80
query38	3003	2861	2785	2785
query39	930	843	860	843
query40	205	130	133	130
query41	54	50	52	50
query42	120	100	107	100
query43	612	582	547	547
query44	1126	753	734	734
query45	191	165	163	163
query46	1077	749	712	712
query47	1880	1772	1798	1772
query48	385	295	306	295
query49	861	431	437	431
query50	773	387	396	387
query51	6843	6833	6719	6719
query52	105	93	97	93
query53	362	298	291	291
query54	886	469	452	452
query55	75	75	75	75
query56	316	291	298	291
query57	1112	1075	1092	1075
query58	259	256	266	256
query59	3434	3266	3267	3266
query60	314	294	298	294
query61	116	115	114	114
query62	589	457	459	457
query63	330	308	296	296
query64	9261	2327	1626	1626
query65	3160	3108	3097	3097
query66	734	326	337	326
query67	15867	14995	15120	14995
query68	5736	546	540	540
query69	707	472	373	373
query70	1206	1130	1157	1130
query71	475	283	290	283
query72	9187	5181	5412	5181
query73	787	323	319	319
query74	5997	5472	5513	5472
query75	4212	2637	2678	2637
query76	3381	1045	951	951
query77	709	322	321	321
query78	13156	9467	8963	8963
query79	6927	541	546	541
query80	1297	491	486	486
query81	595	230	226	226
query82	719	115	106	106
query83	343	173	171	171
query84	275	85	93	85
query85	708	310	303	303
query86	455	308	296	296
query87	3336	3100	3126	3100
query88	5088	2390	2355	2355
query89	506	403	402	402
query90	1964	191	192	191
query91	136	101	107	101
query92	59	49	50	49
query93	4815	524	522	522
query94	1316	210	209	209
query95	419	314	325	314
query96	621	274	266	266
query97	3215	3057	3067	3057
query98	226	204	192	192
query99	1129	833	866	833
Total cold run time: 288309 ms
Total hot run time: 173309 ms

doris-robot · 2024-07-05T10:52:16Z

ClickBench: Total hot run time: 31.08 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 083604df5168f586dd6a5f37c7df475d9c03fba7, data reload: false
query1	0.04	0.03	0.03
query2	0.08	0.04	0.04
query3	0.22	0.05	0.06
query4	1.66	0.09	0.09
query5	0.51	0.51	0.50
query6	1.12	0.72	0.72
query7	0.02	0.02	0.01
query8	0.05	0.04	0.04
query9	0.55	0.50	0.50
query10	0.54	0.55	0.56
query11	0.16	0.12	0.11
query12	0.14	0.12	0.12
query13	0.61	0.59	0.59
query14	0.78	0.77	0.78
query15	0.85	0.81	0.82
query16	0.36	0.37	0.37
query17	0.99	1.03	1.00
query18	0.22	0.27	0.25
query19	1.80	1.66	1.76
query20	0.01	0.00	0.01
query21	15.39	0.76	0.67
query22	4.62	6.38	2.26
query23	18.28	1.42	1.27
query24	2.15	0.23	0.22
query25	0.16	0.08	0.08
query26	0.30	0.22	0.21
query27	0.46	0.23	0.24
query28	13.67	1.01	1.02
query29	12.64	3.31	3.33
query30	0.26	0.07	0.05
query31	2.86	0.38	0.40
query32	3.30	0.48	0.47
query33	2.84	2.91	2.89
query34	17.15	4.35	4.36
query35	4.44	4.43	4.46
query36	0.64	0.48	0.48
query37	0.18	0.15	0.16
query38	0.16	0.14	0.14
query39	0.05	0.03	0.03
query40	0.15	0.12	0.11
query41	0.09	0.06	0.05
query42	0.05	0.04	0.05
query43	0.04	0.04	0.04
Total cold run time: 110.59 s
Total hot run time: 31.08 s

github-actions · 2024-07-08T02:20:08Z

PR approved by at least one committer and no changes requested.

github-actions · 2024-07-08T02:20:11Z

PR approved by anyone and no changes requested.

follow up #37349 1. Some of constants are calculated in FE, it should output same format as in BE. Add `FormatOptions` to control the output format of complex types. 3. Modify some of result files of test cases.

follow up apache#37349 1. Some of constants are calculated in FE, it should output same format as in BE. Add `FormatOptions` to control the output format of complex types. 3. Modify some of result files of test cases.

apache#37349) ## Proposed changes error code ```C++ return creator_without_type::create<AggregateFunctionForEach>(transform_arguments, true, nested_function); ``` "transform_arguments is an internal type of array. All internal types of the array are null, so an array that is not null was mistakenly treated as a null array."

#37798) ## Proposed changes #37349 error code ```C++ return creator_without_type::create<AggregateFunctionForEach>(transform_arguments, true, nested_function); ``` "transform_arguments is an internal type of array. All internal types of the array are null, so an array that is not null was mistakenly treated as a null array." ## Proposed changes Issue Number: close #xxx

#37349) ## Proposed changes error code ```C++ return creator_without_type::create<AggregateFunctionForEach>(transform_arguments, true, nested_function); ``` "transform_arguments is an internal type of array. All internal types of the array are null, so an array that is not null was mistakenly treated as a null array."

follow up #37349 1. Some of constants are calculated in FE, it should output same format as in BE. Add `FormatOptions` to control the output format of complex types. 3. Modify some of result files of test cases.

apache#37349) ## Proposed changes error code ```C++ return creator_without_type::create<AggregateFunctionForEach>(transform_arguments, true, nested_function); ``` "transform_arguments is an internal type of array. All internal types of the array are null, so an array that is not null was mistakenly treated as a null array."

follow up apache#37349 1. Some of constants are calculated in FE, it should output same format as in BE. Add `FormatOptions` to control the output format of complex types. 3. Modify some of result files of test cases.

Mryange added 2 commits July 5, 2024 17:55

fix
39fbd60

rename
083604d

morningman mentioned this pull request Jul 5, 2024
[fix](case) fix struct format out files #37350
Merged

zhangstar333 approved these changes Jul 8, 2024
View reviewed changes

github-actionsBot added approved Indicates a PR has been approved by one committer. reviewed labels Jul 8, 2024

zclllyybb approved these changes Jul 8, 2024
View reviewed changes

zhangstar333 added the dev/2.1.x label Jul 8, 2024

zhangstar333 merged commit 65bf32a into apache:masterJul 8, 2024

HappenLee added the dev/3.0.x label Jul 12, 2024

Mryange mentioned this pull request Jul 15, 2024
[fix](function) fix core when input not null array in foreach functio… #37798
Merged

yiguolei added dev/2.1.5-merged and removed dev/2.1.x labels Jul 16, 2024

dataroaring added dev/3.0.1-merged and removed dev/3.0.x labels Jul 25, 2024

gavinchou mentioned this pull request Aug 19, 2024
Release Note 3.0.1 #39570
Closed

Navigation Menu

Uh oh!

[fix](function) fix core when input not null array in foreach function - #37349

[fix](function) fix core when input not null array in foreach function#37349
zhangstar333 merged 2 commits into
apache:masterfrom
Mryange:foreach-input-not-null

Mryange commented Jul 5, 2024

Uh oh!

doris-robot commented Jul 5, 2024

Uh oh!

Mryange commented Jul 5, 2024

Uh oh!

github-actionsBot commented Jul 5, 2024

Uh oh!

doris-robot commented Jul 5, 2024

Uh oh!

doris-robot commented Jul 5, 2024

Uh oh!

doris-robot commented Jul 5, 2024

Uh oh!

github-actionsBot commented Jul 8, 2024

Uh oh!

github-actionsBot commented Jul 8, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

Uh oh!

Conversation

Mryange commented Jul 5, 2024

Proposed changes

Uh oh!

doris-robot commented Jul 5, 2024

Uh oh!

Mryange commented Jul 5, 2024

Uh oh!

github-actionsBot commented Jul 5, 2024

Uh oh!

doris-robot commented Jul 5, 2024

Uh oh!

doris-robot commented Jul 5, 2024

Uh oh!

doris-robot commented Jul 5, 2024

Uh oh!

github-actionsBot commented Jul 8, 2024

Uh oh!

github-actionsBot commented Jul 8, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants