[FIX](agg) fix vertical_compaction_reader for agg table with array/map type #33130 by amorynan · Pull Request #33216 · apache/doris

amorynan · 2024-04-03T07:54:14Z

Proposed changes

link: #33130
Issue Number: close #xxx

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

doris-robot · 2024-04-03T07:54:19Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

amorynan · 2024-04-03T07:54:23Z

run buildall

github-actions

clang-tidy made some suggestions

doris-robot · 2024-04-03T08:45:15Z

TPC-H: Total hot run time: 50110 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit ad81b1353926812b545ada4f2035aa4308943fa2, data reload: false
------ Round 1 ----------------------------------
q1	17554	4331	4325	4325
q2	2023	157	144	144
q3	10263	1876	1896	1876
q4	10102	1255	1317	1255
q5	8467	4638	4028	4028
q6	229	121	121	121
q7	2061	1620	1629	1620
q8	9306	2742	2733	2733
q9	10884	11362	10523	10523
q10	8670	3529	3547	3529
q11	422	237	244	237
q12	462	295	293	293
q13	18349	3938	4016	3938
q14	340	330	330	330
q15	509	451	450	450
q16	684	580	581	580
q17	1140	974	969	969
q18	7272	6965	6858	6858
q19	1681	1582	1521	1521
q20	524	298	291	291
q21	4456	4101	4101	4101
q22	501	411	388	388
Total cold run time: 115899 ms
Total hot run time: 50110 ms
----- Round 2, with runtime_filter_mode=off -----
q1	4304	4334	4301	4301
q2	320	222	213	213
q3	4153	4147	4137	4137
q4	2735	2760	2736	2736
q5	7335	7215	7200	7200
q6	233	120	115	115
q7	3263	2809	2805	2805
q8	4363	4476	4484	4476
q9	17159	17036	17007	17007
q10	4246	4282	4272	4272
q11	765	682	663	663
q12	1033	875	841	841
q13	6726	3730	3744	3730
q14	447	424	434	424
q15	497	454	447	447
q16	763	698	692	692
q17	3826	3860	3851	3851
q18	8756	8779	8798	8779
q19	1714	1703	1665	1665
q20	2393	2140	2101	2101
q21	8472	8578	8490	8490
q22	1060	989	978	978
Total cold run time: 84563 ms
Total hot run time: 79923 ms

doris-robot · 2024-04-03T08:51:06Z

TeamCity be ut coverage result:
Function Coverage: 37.79% (8051/21303)
Line Coverage: 29.48% (65777/223113)
Region Coverage: 28.94% (33845/116931)
Branch Coverage: 24.81% (17382/70070)
Coverage Report: http://coverage.selectdb-in.cc/coverage/ad81b1353926812b545ada4f2035aa4308943fa2_ad81b1353926812b545ada4f2035aa4308943fa2/report/index.html

doris-robot · 2024-04-03T08:57:58Z

TPC-DS: Total hot run time: 200888 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit ad81b1353926812b545ada4f2035aa4308943fa2, data reload: false
query1	906	394	389	389
query2	6517	2152	2272	2152
query3	6908	199	198	198
query4	19891	18027	17996	17996
query5	19727	6521	6441	6441
query6	285	211	222	211
query7	4156	297	302	297
query8	260	240	247	240
query9	3146	2742	2642	2642
query10	417	284	305	284
query11	11350	10917	10778	10778
query12	123	75	72	72
query13	5575	643	611	611
query14	17616	13083	13270	13083
query15	362	220	245	220
query16	6457	286	256	256
query17	1716	1478	870	870
query18	2312	403	399	399
query19	206	144	147	144
query20	77	75	73	73
query21	187	95	90	90
query22	5405	5122	5126	5122
query23	32738	31828	31848	31828
query24	6942	6522	6501	6501
query25	521	437	405	405
query26	507	162	159	159
query27	1794	293	289	289
query28	6166	2299	2264	2264
query29	2985	2835	2808	2808
query30	235	160	160	160
query31	907	755	710	710
query32	67	60	57	57
query33	400	240	246	240
query34	842	452	479	452
query35	1120	851	932	851
query36	1261	1158	1365	1158
query37	87	59	58	58
query38	3132	2992	2940	2940
query39	1389	1322	1309	1309
query40	201	97	92	92
query41	36	34	35	34
query42	84	78	80	78
query43	609	645	611	611
query44	1112	728	725	725
query45	239	224	224	224
query46	1254	976	937	937
query47	1903	1589	1694	1589
query48	957	686	672	672
query49	624	367	368	367
query50	848	606	560	560
query51	4777	4675	4600	4600
query52	92	74	75	74
query53	449	307	315	307
query54	2616	2442	2457	2442
query55	93	85	80	80
query56	219	205	203	203
query57	1202	1109	1035	1035
query58	213	211	196	196
query59	3658	3250	3380	3250
query60	215	192	213	192
query61	87	90	83	83
query62	834	456	457	456
query63	465	333	330	330
query64	2243	1465	1387	1387
query65	3641	3525	3569	3525
query66	771	374	362	362
query67	16094	17209	15067	15067
query68	9159	662	682	662
query69	576	340	336	336
query70	1504	1401	1583	1401
query71	392	310	306	306
query72	6481	3444	3468	3444
query73	724	321	320	320
query74	6356	5904	5821	5821
query75	5397	3800	3772	3772
query76	5760	1181	1204	1181
query77	972	258	245	245
query78	12583	11890	11413	11413
query79	12856	658	640	640
query80	1590	400	393	393
query81	492	229	232	229
query82	1221	101	93	93
query83	161	129	127	127
query84	255	68	68	68
query85	873	291	278	278
query86	338	300	294	294
query87	3258	3025	3034	3025
query88	5520	2384	2351	2351
query89	433	295	271	271
query90	1960	203	209	203
query91	150	120	116	116
query92	55	52	52	52
query93	6130	584	557	557
query94	753	211	203	203
query95	1113	1069	1081	1069
query96	660	338	325	325
query97	6503	6438	6404	6404
query98	186	172	166	166
query99	3093	983	888	888
Total cold run time: 317732 ms
Total hot run time: 200888 ms

doris-robot · 2024-04-03T09:03:26Z

ClickBench: Total hot run time: 30.72 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit ad81b1353926812b545ada4f2035aa4308943fa2, data reload: false
query1	0.03	0.02	0.02
query2	0.07	0.02	0.02
query3	0.25	0.06	0.06
query4	1.77	0.07	0.08
query5	0.53	0.52	0.52
query6	1.25	0.66	0.62
query7	0.01	0.01	0.01
query8	0.04	0.02	0.02
query9	0.55	0.51	0.51
query10	0.56	0.56	0.55
query11	0.13	0.09	0.09
query12	0.12	0.09	0.09
query13	0.62	0.61	0.61
query14	0.78	0.79	0.81
query15	0.78	0.76	0.76
query16	0.37	0.37	0.37
query17	1.03	1.00	1.03
query18	0.23	0.25	0.26
query19	1.90	1.76	1.80
query20	0.02	0.01	0.02
query21	15.48	0.62	0.55
query22	2.13	2.34	1.52
query23	17.82	1.07	0.90
query24	5.49	1.15	1.09
query25	0.38	0.14	0.05
query26	0.56	0.16	0.16
query27	0.05	0.04	0.03
query28	7.43	0.73	0.74
query29	12.61	2.26	2.28
query30	0.62	0.58	0.57
query31	2.83	0.39	0.38
query32	3.37	0.50	0.50
query33	3.11	3.00	3.12
query34	15.23	4.78	4.81
query35	4.85	4.82	4.86
query36	1.07	1.01	1.02
query37	0.06	0.05	0.04
query38	0.04	0.02	0.02
query39	0.02	0.01	0.02
query40	0.17	0.14	0.14
query41	0.06	0.01	0.01
query42	0.02	0.01	0.01
query43	0.03	0.02	0.01
Total cold run time: 104.47 s
Total hot run time: 30.72 s

doris-robot · 2024-04-03T09:06:39Z

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit ad81b1353926812b545ada4f2035aa4308943fa2 with default session variables
Stream load json: 20 seconds loaded 2358488459 Bytes, about 112 MB/s
Stream load orc: 59 seconds loaded 1101869774 Bytes, about 17 MB/s
Stream load parquet: 31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select: 20.1 seconds inserted 10000000 Rows, about 497K ops/s

amorynan · 2024-04-03T09:44:02Z

run buildall

doris-robot · 2024-04-03T10:02:47Z

TPC-H: Total hot run time: 50054 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit bf99356163c5f10379891b3c52691493d2954d82, data reload: false
------ Round 1 ----------------------------------
q1	17609	4435	4376	4376
q2	2026	156	141	141
q3	10274	1947	1936	1936
q4	10098	1249	1343	1249
q5	8437	4156	3973	3973
q6	239	122	122	122
q7	2053	1592	1613	1592
q8	9266	2745	2756	2745
q9	10798	10756	10394	10394
q10	8640	3550	3504	3504
q11	421	242	239	239
q12	472	296	299	296
q13	18333	3926	4004	3926
q14	358	327	323	323
q15	493	460	458	458
q16	688	601	604	601
q17	1135	970	978	970
q18	7269	6796	7012	6796
q19	1686	1577	1540	1540
q20	532	315	299	299
q21	4489	4176	4189	4176
q22	505	415	398	398
Total cold run time: 115821 ms
Total hot run time: 50054 ms
----- Round 2, with runtime_filter_mode=off -----
q1	4337	4299	4280	4280
q2	320	220	220	220
q3	4144	4157	4168	4157
q4	2761	2744	2768	2744
q5	7241	7231	7213	7213
q6	238	118	118	118
q7	3286	2810	2857	2810
q8	4321	4468	4512	4468
q9	17287	17040	17199	17040
q10	4253	4292	4290	4290
q11	767	681	671	671
q12	1038	853	836	836
q13	7309	3734	3724	3724
q14	455	420	424	420
q15	499	445	454	445
q16	761	708	704	704
q17	3843	3830	3797	3797
q18	8910	8868	8890	8868
q19	1723	1723	1636	1636
q20	2380	2179	2091	2091
q21	8572	8518	8389	8389
q22	1082	949	927	927
Total cold run time: 85527 ms
Total hot run time: 79848 ms

doris-robot · 2024-04-03T10:09:22Z

TeamCity be ut coverage result:
Function Coverage: 37.78% (8048/21303)
Line Coverage: 29.47% (65754/223122)
Region Coverage: 28.93% (33830/116936)
Branch Coverage: 24.80% (17375/70074)
Coverage Report: http://coverage.selectdb-in.cc/coverage/bf99356163c5f10379891b3c52691493d2954d82_bf99356163c5f10379891b3c52691493d2954d82/report/index.html

doris-robot · 2024-04-03T10:15:19Z

TPC-DS: Total hot run time: 201177 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit bf99356163c5f10379891b3c52691493d2954d82, data reload: false
query1	914	390	382	382
query2	6543	2198	2055	2055
query3	6913	195	197	195
query4	20468	17871	17981	17871
query5	19714	6593	6548	6548
query6	289	219	233	219
query7	4157	304	291	291
query8	277	274	269	269
query9	3138	2712	2629	2629
query10	427	286	306	286
query11	11414	10689	10734	10689
query12	115	78	71	71
query13	5575	638	637	637
query14	17889	13705	13305	13305
query15	359	226	233	226
query16	6456	271	257	257
query17	1740	1438	857	857
query18	2332	395	395	395
query19	207	147	138	138
query20	78	75	75	75
query21	182	95	88	88
query22	5192	5009	4995	4995
query23	32564	31840	31883	31840
query24	7678	6599	6512	6512
query25	521	414	414	414
query26	816	162	155	155
query27	2195	292	287	287
query28	6159	2270	2236	2236
query29	2797	2702	2650	2650
query30	238	165	160	160
query31	901	704	723	704
query32	63	62	58	58
query33	400	244	243	243
query34	850	459	506	459
query35	1100	934	935	934
query36	1357	1122	1180	1122
query37	93	59	59	59
query38	3095	2866	2896	2866
query39	1366	1327	1314	1314
query40	310	91	92	91
query41	35	33	32	32
query42	85	83	80	80
query43	633	652	629	629
query44	1167	726	728	726
query45	238	226	222	222
query46	1249	966	958	958
query47	1835	1669	1639	1639
query48	966	673	651	651
query49	655	360	369	360
query50	876	601	595	595
query51	4764	4665	4660	4660
query52	83	72	83	72
query53	436	321	315	315
query54	2639	2432	2443	2432
query55	89	77	79	77
query56	204	206	188	188
query57	1192	1125	1131	1125
query58	212	185	182	182
query59	3352	3282	3192	3192
query60	206	192	194	192
query61	85	83	87	83
query62	820	492	466	466
query63	465	327	320	320
query64	2818	1535	1296	1296
query65	3638	3544	3543	3543
query66	781	366	363	363
query67	16072	15191	15834	15191
query68	8173	641	686	641
query69	567	341	341	341
query70	1536	1619	1545	1545
query71	385	311	310	310
query72	6221	3379	3458	3379
query73	733	325	323	323
query74	6296	5910	5916	5910
query75	4506	3747	3804	3747
query76	4586	1113	1221	1113
query77	550	242	241	241
query78	12471	12510	11831	11831
query79	10536	641	654	641
query80	1165	392	390	390
query81	501	231	230	230
query82	656	95	95	95
query83	165	134	133	133
query84	256	67	67	67
query85	899	278	274	274
query86	346	315	291	291
query87	3187	3021	3025	3021
query88	4837	2356	2361	2356
query89	410	311	274	274
query90	1786	202	199	199
query91	146	119	116	116
query92	60	51	48	48
query93	4682	557	612	557
query94	785	204	200	200
query95	1107	1057	1029	1029
query96	632	332	319	319
query97	6468	6321	6355	6321
query98	187	163	165	163
query99	3011	901	916	901
Total cold run time: 310323 ms
Total hot run time: 201177 ms

doris-robot · 2024-04-03T10:20:46Z

ClickBench: Total hot run time: 30.99 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit bf99356163c5f10379891b3c52691493d2954d82, data reload: false
query1	0.02	0.02	0.02
query2	0.07	0.02	0.02
query3	0.25	0.04	0.06
query4	1.79	0.07	0.06
query5	0.54	0.53	0.51
query6	1.24	0.60	0.61
query7	0.01	0.01	0.01
query8	0.03	0.02	0.02
query9	0.54	0.47	0.49
query10	0.54	0.55	0.53
query11	0.12	0.08	0.08
query12	0.12	0.09	0.08
query13	0.61	0.60	0.61
query14	0.78	0.77	0.79
query15	0.78	0.76	0.75
query16	0.37	0.40	0.38
query17	1.02	1.02	1.00
query18	0.24	0.25	0.25
query19	1.91	1.82	1.77
query20	0.02	0.01	0.01
query21	15.48	0.55	0.54
query22	1.98	2.29	2.16
query23	17.23	0.97	1.08
query24	6.48	0.84	1.30
query25	0.39	0.12	0.05
query26	0.67	0.16	0.15
query27	0.04	0.04	0.04
query28	6.53	0.77	0.72
query29	12.65	2.23	2.31
query30	0.57	0.53	0.53
query31	2.82	0.39	0.37
query32	3.39	0.50	0.50
query33	3.09	3.05	3.04
query34	15.24	4.78	4.82
query35	4.85	4.84	4.83
query36	1.10	1.02	1.02
query37	0.06	0.05	0.05
query38	0.04	0.02	0.02
query39	0.01	0.01	0.01
query40	0.16	0.15	0.14
query41	0.07	0.02	0.01
query42	0.02	0.01	0.01
query43	0.02	0.01	0.02
Total cold run time: 103.89 s
Total hot run time: 30.99 s

doris-robot · 2024-04-03T10:24:00Z

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit bf99356163c5f10379891b3c52691493d2954d82 with default session variables
Stream load json: 20 seconds loaded 2358488459 Bytes, about 112 MB/s
Stream load orc: 58 seconds loaded 1101869774 Bytes, about 18 MB/s
Stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
Insert into select: 20.0 seconds inserted 10000000 Rows, about 500K ops/s

…p type apache#33130 (apache#33216)

amorynan added 3 commits April 3, 2024 15:47

fix vertical_compaction_reader for agg table with array/map type
69fce80

fix dcheck and normalize function
dd39852

fixed
ad81b13

github-actionsBot added the kind/test label Apr 3, 2024

github-actionsBot reviewed Apr 3, 2024
View reviewed changes

Comment threadbe/src/vec/columns/column.h
Comment threadbe/src/vec/columns/column_array.h
Comment threadbe/src/vec/columns/column_const.h
Comment threadbe/src/vec/columns/column_map.h
Comment threadbe/src/vec/columns/column_nullable.h
Comment threadbe/src/vec/columns/column_string.h

xiaokang changed the title ~~[FIX](agg)Fix vertical blocker reader with array~~[FIX](agg) fix vertical_compaction_reader for agg table with array/map type #33130Apr 3, 2024

fix test
bf99356

xiaokang merged commit dbd1da4 into apache:branch-2.0Apr 3, 2024

mongo360 pushed a commit to mongo360/doris that referenced this pull request Aug 16, 2024
[FIX](agg) fix vertical_compaction_reader for agg table with array/ma…
0198024
…p type apache#33130 (apache#33216)

Uh oh!

Conversation

amorynan commented Apr 3, 2024

Proposed changes

Further comments

Uh oh!

doris-robot commented Apr 3, 2024

Uh oh!

amorynan commented Apr 3, 2024

Uh oh!

github-actionsBot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

doris-robot commented Apr 3, 2024

Uh oh!

doris-robot commented Apr 3, 2024

Uh oh!

doris-robot commented Apr 3, 2024

Uh oh!

doris-robot commented Apr 3, 2024

Uh oh!

doris-robot commented Apr 3, 2024

Uh oh!

amorynan commented Apr 3, 2024

Uh oh!

doris-robot commented Apr 3, 2024

Uh oh!

doris-robot commented Apr 3, 2024

Uh oh!

doris-robot commented Apr 3, 2024

Uh oh!

doris-robot commented Apr 3, 2024

Uh oh!

doris-robot commented Apr 3, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants