Skip to content

[fix](simdjson) fix simdjson reader will parse given root twice when data is object array - #38490

Merged
xiaokang merged 1 commit into
apache:masterfrom
amorynan:fix-simdjson-reader-multi-parse-root
Jul 30, 2024
Merged

[fix](simdjson) fix simdjson reader will parse given root twice when data is object array #38490
xiaokang merged 1 commit into
apache:masterfrom
amorynan:fix-simdjson-reader-multi-parse-root

Conversation

@amorynan

@amorynanamorynan commented Jul 29, 2024

Copy link
Copy Markdown
Contributor

Proposed changes

if we set jsonroot is not empty , and data is object array , here will meet status like this:
image

Issue Number: close #xxx

@doris-robot

doris-robot commented Jul 29, 2024

Copy link
Copy Markdown

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@amorynan

amorynan commented Jul 29, 2024

Copy link
Copy Markdown
ContributorAuthor

run buildall

@github-actions

github-actionsBot commented Jul 29, 2024

Copy link
Copy Markdown
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@doris-robot

doris-robot commented Jul 29, 2024

Copy link
Copy Markdown
TPC-H: Total hot run time: 42326 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit d4bb5fa642fee8c2280573d622ea47279c3b928d, data reload: false
------ Round 1 ----------------------------------
q1	17627	4469	4238	4238
q2	2040	205	205	205
q3	10642	1335	1390	1335
q4	10582	887	969	887
q5	7973	3060	3024	3024
q6	226	141	142	141
q7	1068	642	653	642
q8	9944	1998	1991	1991
q9	8741	6724	6728	6724
q10	9775	3893	3874	3874
q11	436	260	260	260
q12	410	242	234	234
q13	17772	2921	2970	2921
q14	269	248	250	248
q15	539	498	504	498
q16	524	414	396	396
q17	992	949	912	912
q18	8086	7274	7213	7213
q19	1397	1210	1222	1210
q20	563	333	346	333
q21	5316	4747	4808	4747
q22	355	295	293	293
Total cold run time: 115277 ms
Total hot run time: 42326 ms
----- Round 2, with runtime_filter_mode=off -----
q1	4178	4074	4090	4074
q2	336	224	228	224
q3	3052	2996	3021	2996
q4	1891	1896	1883	1883
q5	5268	5241	5242	5241
q6	219	135	134	134
q7	2102	1690	1693	1690
q8	3241	3294	3302	3294
q9	8323	8355	8290	8290
q10	3829	3901	3894	3894
q11	546	453	456	453
q12	714	561	530	530
q13	13528	2907	2967	2907
q14	299	258	266	258
q15	538	488	485	485
q16	438	408	406	406
q17	1768	1715	1691	1691
q18	7770	7545	7142	7142
q19	1665	1676	1660	1660
q20	1980	1823	1769	1769
q21	5550	5179	5234	5179
q22	526	465	466	465
Total cold run time: 67761 ms
Total hot run time: 54665 ms

@doris-robot

doris-robot commented Jul 29, 2024

Copy link
Copy Markdown
TPC-DS: Total hot run time: 169715 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit d4bb5fa642fee8c2280573d622ea47279c3b928d, data reload: false
query1	912	369	364	364
query2	6465	1651	1650	1650
query3	6667	213	233	213
query4	19039	17453	17218	17218
query5	4262	528	527	527
query6	295	169	185	169
query7	4608	300	304	300
query8	265	193	187	187
query9	8517	2425	2414	2414
query10	462	293	275	275
query11	10492	10103	9888	9888
query12	138	94	87	87
query13	1628	393	373	373
query14	8841	8015	7838	7838
query15	221	163	167	163
query16	7037	442	450	442
query17	970	608	560	560
query18	1884	291	292	291
query19	195	148	147	147
query20	96	84	87	84
query21	210	112	110	110
query22	4344	3913	3863	3863
query23	33779	33209	33045	33045
query24	10401	3095	3193	3095
query25	705	408	407	407
query26	1777	157	160	157
query27	3024	279	286	279
query28	6999	2016	2002	2002
query29	1260	448	435	435
query30	287	154	151	151
query31	938	759	761	759
query32	102	59	59	59
query33	714	322	322	322
query34	916	485	508	485
query35	850	740	728	728
query36	983	892	888	888
query37	279	80	83	80
query38	2853	2755	2781	2755
query39	865	806	821	806
query40	285	121	120	120
query41	51	47	52	47
query42	121	101	105	101
query43	471	438	430	430
query44	1196	730	729	729
query45	211	178	180	178
query46	1085	807	804	804
query47	1796	1677	1690	1677
query48	368	305	304	304
query49	1185	440	438	438
query50	896	446	446	446
query51	6691	6665	6571	6571
query52	111	92	94	92
query53	255	180	182	180
query54	635	461	476	461
query55	77	78	78	78
query56	307	271	265	265
query57	1143	1032	1041	1032
query58	311	299	381	299
query59	2464	2465	2361	2361
query60	324	282	276	276
query61	102	100	100	100
query62	913	656	657	656
query63	215	189	194	189
query64	5914	1935	1923	1923
query65	3156	3095	3093	3093
query66	1442	351	338	338
query67	15399	14809	14648	14648
query68	4504	608	596	596
query69	678	381	310	310
query70	1115	1022	1069	1022
query71	482	281	274	274
query72	7972	2706	2538	2538
query73	773	336	332	332
query74	5990	5547	5667	5547
query75	4132	2730	2707	2707
query76	3088	1328	1422	1328
query77	715	310	317	310
query78	9621	8914	8889	8889
query79	2886	538	542	538
query80	1298	542	501	501
query81	576	229	233	229
query82	792	128	131	128
query83	380	171	174	171
query84	279	82	82	82
query85	1797	347	301	301
query86	424	295	312	295
query87	3282	3086	3115	3086
query88	3857	2478	2495	2478
query89	409	291	287	287
query90	2024	190	198	190
query91	133	99	107	99
query92	62	48	49	48
query93	3858	639	628	628
query94	1028	304	309	304
query95	388	277	274	274
query96	617	282	282	282
query97	3174	3060	3023	3023
query98	226	195	187	187
query99	1608	1325	1279	1279
Total cold run time: 269572 ms
Total hot run time: 169715 ms

@doris-robot

doris-robot commented Jul 29, 2024

Copy link
Copy Markdown
ClickBench: Total hot run time: 30.04 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit d4bb5fa642fee8c2280573d622ea47279c3b928d, data reload: false
query1	0.04	0.03	0.04
query2	0.08	0.04	0.04
query3	0.22	0.05	0.06
query4	1.67	0.07	0.07
query5	0.49	0.47	0.49
query6	1.15	0.72	0.71
query7	0.02	0.02	0.02
query8	0.06	0.04	0.05
query9	0.58	0.51	0.51
query10	0.56	0.56	0.56
query11	0.15	0.11	0.11
query12	0.14	0.12	0.12
query13	0.61	0.59	0.60
query14	0.76	0.78	0.80
query15	0.88	0.85	0.85
query16	0.35	0.37	0.36
query17	0.99	0.96	1.03
query18	0.23	0.21	0.21
query19	1.85	1.73	1.77
query20	0.01	0.01	0.00
query21	15.41	0.78	0.67
query22	4.56	7.68	1.38
query23	18.00	1.37	1.28
query24	2.24	0.23	0.23
query25	0.19	0.09	0.09
query26	0.31	0.21	0.21
query27	0.46	0.23	0.24
query28	13.17	1.00	0.97
query29	12.52	3.33	3.34
query30	0.25	0.06	0.05
query31	2.85	0.39	0.40
query32	3.26	0.48	0.49
query33	2.95	2.93	2.92
query34	15.43	4.26	4.24
query35	4.24	4.31	4.29
query36	0.66	0.48	0.48
query37	0.20	0.16	0.16
query38	0.16	0.14	0.16
query39	0.05	0.04	0.04
query40	0.15	0.12	0.12
query41	0.10	0.05	0.05
query42	0.06	0.05	0.05
query43	0.04	0.03	0.05
Total cold run time: 108.1 s
Total hot run time: 30.04 s

@amorynanamorynan mentioned this pull request Jul 29, 2024
Merged
eldenmoon
eldenmoon approved these changes Jul 30, 2024
View reviewed changes

@eldenmooneldenmoon left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@xiaokangxiaokang added the usercase Important user case type label label Jul 30, 2024
xiaokang
xiaokang approved these changes Jul 30, 2024
View reviewed changes

@xiaokangxiaokang left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

xiaokang pushed a commit that referenced this pull request Jul 30, 2024
amory
[fix](simdjson) fix simdjson reader for read json object array when j…
973f287
…sonroot set #38490 (#38500)
@github-actionsgithub-actionsBot added the approved Indicates a PR has been approved by one committer. label Jul 30, 2024
@github-actions

github-actionsBot commented Jul 30, 2024

Copy link
Copy Markdown
Contributor

PR approved by at least one committer and no changes requested.

@github-actions

github-actionsBot commented Jul 30, 2024

Copy link
Copy Markdown
Contributor

PR approved by anyone and no changes requested.

@xiaokang
xiaokang merged commit 15eac46 into apache:masterJul 30, 2024
dataroaring pushed a commit that referenced this pull request Jul 31, 2024
amory@dataroaring
[fix](simdjson) fix simdjson reader will parse given root twice when …
1e13f0d
…data is object array (#38490)
@amorynanamorynan mentioned this pull request Jul 31, 2024
Merged
yiguolei pushed a commit that referenced this pull request Aug 1, 2024
amory
[pick](simdjson) fix simdjson with object array when jsonroot is not …
338fa32
…empty (#38633)
## Proposed changes
backport: #38490
Issue Number: close #xxx
<!--Describe your changes.-->
@xiaokangxiaokang removed the doing label Aug 6, 2024
@gavinchougavinchou mentioned this pull request Aug 19, 2024
Closed
@yiguoleiyiguolei mentioned this pull request Sep 5, 2024
Closed
3 tasks
mongo360 pushed a commit to mongo360/doris that referenced this pull request Dec 11, 2024
amorycaokaihua1
[fix](simdjson) fix simdjson reader for read json object array when j…
1b6fab1
…sonroot set apache#38490 (apache#38500)
HappenLee pushed a commit to HappenLee/incubator-doris that referenced this pull request Apr 24, 2026
amory@HappenLee
[fix](simdjson) fix simdjson reader will parse given root twice when …
f4cbd12
…data is object array (apache#38490)
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@xiaokangxiaokangxiaokang approved these changes
@eldenmooneldenmooneldenmoon approved these changes

Assignees

No one assigned

Labels

approvedIndicates a PR has been approved by one committer.dev/2.0.14-mergeddev/2.1.6-mergeddev/3.0.1-mergedreviewedusercaseImportant user case type label

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

6 participants

@amorynan@doris-robot@xiaokang@eldenmoon@yiguolei@dataroaring