Skip to content

[fix](ES Catalog)Support parse single value for array column (#40614) - #40666

Merged
xiaokang merged 1 commit into
apache:branch-2.0from
qidaye:pick_fix_array_and_string_2.0
Sep 11, 2024
Merged

[fix](ES Catalog)Support parse single value for array column (#40614)#40666
xiaokang merged 1 commit into
apache:branch-2.0from
qidaye:pick_fix_array_and_string_2.0

Conversation

@qidaye

@qidayeqidaye commented Sep 11, 2024

Copy link
Copy Markdown
Contributor

@doris-robot

doris-robot commented Sep 11, 2024

Copy link
Copy Markdown

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@qidaye

qidaye commented Sep 11, 2024

Copy link
Copy Markdown
ContributorAuthor

run buildall

@github-actions

github-actionsBot commented Sep 11, 2024

Copy link
Copy Markdown
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@qidaye
qidayeforce-pushed the pick_fix_array_and_string_2.0 branch from 5e47e46 to e2bb504CompareSeptember 11, 2024 07:29
@qidaye

qidaye commented Sep 11, 2024

Copy link
Copy Markdown
ContributorAuthor

run buildall

@qidaye
[fix](ES Catalog)Support parse single value for array column (apache#…
7afd313
…40614)
Follow up apache#39104, when the field has one value and we map it as array
type in Doris, we parse the single value to a single element array to
make them queryable.
closeapache#40406
@qidaye
qidayeforce-pushed the pick_fix_array_and_string_2.0 branch from e2bb504 to 7afd313CompareSeptember 11, 2024 07:31
@qidaye

qidaye commented Sep 11, 2024

Copy link
Copy Markdown
ContributorAuthor

run buildall

@github-actions

github-actionsBot commented Sep 11, 2024

Copy link
Copy Markdown
Contributor

clang-tidy review says "All clean, LGTM! 👍"

1 similar comment
@github-actions

github-actionsBot commented Sep 11, 2024

Copy link
Copy Markdown
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@doris-robot

doris-robot commented Sep 11, 2024

Copy link
Copy Markdown
TPC-H: Total hot run time: 49143 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 7afd313c5b96fecc620283af738b83167e64a245, data reload: false
------ Round 1 ----------------------------------
q1	17651	4411	4347	4347
q2	2095	157	145	145
q3	10335	1901	1877	1877
q4	10115	1236	1333	1236
q5	8523	3956	3932	3932
q6	229	124	124	124
q7	2030	1603	1604	1603
q8	9540	2715	2710	2710
q9	13736	9859	9841	9841
q10	8613	3531	3529	3529
q11	411	255	253	253
q12	466	303	306	303
q13	18344	3975	3983	3975
q14	356	326	338	326
q15	507	461	461	461
q16	539	446	467	446
q17	1145	969	968	968
q18	7201	6899	6775	6775
q19	1656	1569	1521	1521
q20	528	312	308	308
q21	4434	4104	4078	4078
q22	495	389	385	385
Total cold run time: 118949 ms
Total hot run time: 49143 ms
----- Round 2, with runtime_filter_mode=off -----
q1	4348	4273	4316	4273
q2	326	226	234	226
q3	4167	4137	4100	4100
q4	2751	2752	2769	2752
q5	7184	7161	7128	7128
q6	239	120	121	120
q7	3228	2885	2824	2824
q8	4375	4523	4496	4496
q9	14196	13992	13878	13878
q10	4214	4264	4217	4217
q11	770	696	678	678
q12	1018	839	875	839
q13	7241	3754	3723	3723
q14	455	428	438	428
q15	500	463	458	458
q16	634	622	587	587
q17	3823	3835	3892	3835
q18	8789	8790	8809	8790
q19	1712	1673	1664	1664
q20	2386	2128	2149	2128
q21	8631	8479	8580	8479
q22	1037	920	925	920
Total cold run time: 82024 ms
Total hot run time: 76543 ms

@doris-robot

doris-robot commented Sep 11, 2024

Copy link
Copy Markdown

TeamCity be ut coverage result:
Function Coverage: 37.84% (8141/21513)
Line Coverage: 29.59% (66993/226413)
Region Coverage: 29.07% (34564/118880)
Branch Coverage: 24.98% (17808/71288)
Coverage Report: http://coverage.selectdb-in.cc/coverage/7afd313c5b96fecc620283af738b83167e64a245_7afd313c5b96fecc620283af738b83167e64a245/report/index.html

@doris-robot

doris-robot commented Sep 11, 2024

Copy link
Copy Markdown
TPC-DS: Total hot run time: 212618 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 7afd313c5b96fecc620283af738b83167e64a245, data reload: false
query1	928	403	411	403
query2	6550	2124	2253	2124
query3	6918	221	210	210
query4	23003	21687	21550	21550
query5	19745	6503	6437	6437
query6	276	219	231	219
query7	4330	297	300	297
query8	255	266	232	232
query9	3057	2615	2565	2565
query10	468	314	326	314
query11	15325	15149	15470	15149
query12	127	76	75	75
query13	1036	446	442	442
query14	17209	13145	13537	13145
query15	375	219	232	219
query16	6474	287	262	262
query17	1758	924	913	913
query18	887	326	329	326
query19	212	153	156	153
query20	83	77	77	77
query21	190	98	96	96
query22	5328	4997	5046	4997
query23	34152	33488	33476	33476
query24	7828	6334	6318	6318
query25	521	431	441	431
query26	1260	166	167	166
query27	2392	297	296	296
query28	6101	2255	2217	2217
query29	2825	2792	2726	2726
query30	240	176	169	169
query31	980	759	738	738
query32	74	67	62	62
query33	460	261	266	261
query34	849	484	464	464
query35	1168	943	963	943
query36	1240	1230	1076	1076
query37	176	63	58	58
query38	3057	2893	2941	2893
query39	1359	1327	1338	1327
query40	310	95	95	95
query41	50	37	38	37
query42	91	90	83	83
query43	599	597	552	552
query44	1175	725	722	722
query45	243	227	229	227
query46	1234	964	961	961
query47	2037	1948	1633	1633
query48	501	422	405	405
query49	652	377	383	377
query50	856	635	580	580
query51	4737	4634	4664	4634
query52	90	80	74	74
query53	232	186	184	184
query54	2658	2489	2506	2489
query55	91	90	84	84
query56	232	206	212	206
query57	1246	1034	1117	1034
query58	232	225	226	225
query59	3446	3305	3347	3305
query60	222	210	220	210
query61	102	97	100	97
query62	813	499	458	458
query63	205	170	174	170
query64	3629	1610	1494	1494
query65	3604	3576	3560	3560
query66	777	418	400	400
query67	15943	15791	15088	15088
query68	10934	659	655	655
query69	497	260	277	260
query70	1977	1629	1532	1532
query71	405	307	309	307
query72	6741	4904	4889	4889
query73	766	330	324	324
query74	6392	5898	5856	5856
query75	5297	3681	3616	3616
query76	6593	1146	1234	1146
query77	1156	258	258	258
query78	12871	11621	11896	11621
query79	6935	645	625	625
query80	1234	379	384	379
query81	458	239	240	239
query82	889	98	94	94
query83	173	142	139	139
query84	259	71	72	71
query85	883	330	334	330
query86	333	295	285	285
query87	3210	3018	3005	3005
query88	3982	2401	2409	2401
query89	480	286	285	285
query90	1934	222	220	220
query91	178	131	140	131
query92	63	52	52	52
query93	6710	580	596	580
query94	699	212	212	212
query95	1943	2072	2029	2029
query96	649	336	331	331
query97	6494	6395	6437	6395
query98	224	205	203	203
query99	2991	938	972	938
Total cold run time: 319573 ms
Total hot run time: 212618 ms

@doris-robot

doris-robot commented Sep 11, 2024

Copy link
Copy Markdown
ClickBench: Total hot run time: 30.8 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 7afd313c5b96fecc620283af738b83167e64a245, data reload: false
query1	0.02	0.02	0.02
query2	0.07	0.03	0.02
query3	0.25	0.05	0.05
query4	1.79	0.06	0.05
query5	0.54	0.52	0.53
query6	1.24	0.61	0.61
query7	0.01	0.00	0.01
query8	0.04	0.03	0.02
query9	0.51	0.50	0.51
query10	0.54	0.53	0.54
query11	0.12	0.09	0.09
query12	0.12	0.09	0.09
query13	0.63	0.62	0.61
query14	0.81	0.79	0.80
query15	0.79	0.76	0.76
query16	0.35	0.40	0.38
query17	1.02	1.00	1.01
query18	0.24	0.24	0.25
query19	1.91	1.83	1.84
query20	0.02	0.01	0.01
query21	15.45	0.55	0.55
query22	1.95	2.49	1.58
query23	17.18	0.93	0.96
query24	4.73	1.10	1.46
query25	0.33	0.09	0.05
query26	0.68	0.16	0.15
query27	0.05	0.05	0.03
query28	7.80	0.84	0.72
query29	12.74	2.29	2.26
query30	0.62	0.53	0.51
query31	2.80	0.39	0.37
query32	3.38	0.51	0.49
query33	3.04	3.11	3.05
query34	15.25	4.80	4.80
query35	4.88	4.83	4.86
query36	1.05	1.02	1.02
query37	0.07	0.04	0.04
query38	0.04	0.02	0.02
query39	0.02	0.02	0.01
query40	0.16	0.13	0.15
query41	0.06	0.01	0.02
query42	0.02	0.02	0.02
query43	0.03	0.02	0.01
Total cold run time: 103.35 s
Total hot run time: 30.8 s

@doris-robot

doris-robot commented Sep 11, 2024

Copy link
Copy Markdown

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit 7afd313c5b96fecc620283af738b83167e64a245 with default session variables
Stream load json: 20 seconds loaded 2358488459 Bytes, about 112 MB/s
Stream load orc: 58 seconds loaded 1101869774 Bytes, about 18 MB/s
Stream load parquet: 31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select: 21.5 seconds inserted 10000000 Rows, about 465K ops/s

@xiaokang
xiaokang merged commit c63359a into apache:branch-2.0Sep 11, 2024
@qidaye
qidaye deleted the pick_fix_array_and_string_2.0 branch September 11, 2024 12:47
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

kind/test

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@qidaye@doris-robot@xiaokang