[Fix](JsonReader) Return correct status when parse failed by eldenmoon · Pull Request #39206 · apache/doris

eldenmoon · 2024-08-12T02:49:35Z

When using JsonFunctions::extract_from_object, but input obj is not object type but other types like null, then Status should be simdjson::INCORRECT_TYPE to fill the default value in the later process. Example, json path is $.city.name, but input json is {"city" : null} then Status::NotFound should be returned, in the following, column will be filled with default values.

Proposed changes

Issue Number: close #xxx

doris-robot · 2024-08-12T02:49:39Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

eldenmoon · 2024-08-12T02:49:59Z

run buildall

eldenmoon · 2024-08-12T02:52:02Z

run buildall

github-actions · 2024-08-12T02:56:20Z

clang-tidy review says "All clean, LGTM! 👍"

github-actions · 2024-08-12T02:59:42Z

clang-tidy review says "All clean, LGTM! 👍"

xiaokang

LGTM

github-actions · 2024-08-12T03:26:59Z

PR approved by at least one committer and no changes requested.

github-actions · 2024-08-12T03:27:01Z

PR approved by anyone and no changes requested.

doris-robot · 2024-08-12T03:30:48Z

TPC-H: Total hot run time: 38868 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit e94357a527392964a7364c98d44cb3d0dc236525, data reload: false
------ Round 1 ----------------------------------
q1	17608	4482	4260	4260
q2	2013	204	194	194
q3	11761	1040	1032	1032
q4	10524	746	668	668
q5	7497	2506	2476	2476
q6	230	141	138	138
q7	974	601	593	593
q8	9409	2035	1880	1880
q9	9574	6511	6511	6511
q10	7039	2180	2113	2113
q11	470	240	237	237
q12	389	210	211	210
q13	17745	2967	2989	2967
q14	284	233	237	233
q15	521	491	485	485
q16	499	407	389	389
q17	964	697	724	697
q18	7922	7360	7310	7310
q19	4976	1019	1027	1019
q20	702	322	317	317
q21	5313	4449	4149	4149
q22	1103	1011	990	990
Total cold run time: 117517 ms
Total hot run time: 38868 ms
----- Round 2, with runtime_filter_mode=off -----
q1	4360	4236	4221	4221
q2	373	274	262	262
q3	2850	2604	2770	2604
q4	2001	1716	1684	1684
q5	5569	5548	5426	5426
q6	218	131	128	128
q7	2103	1747	1763	1747
q8	3252	3402	3391	3391
q9	8751	8708	8871	8708
q10	3491	3327	3229	3229
q11	598	478	498	478
q12	784	622	618	618
q13	16003	3157	3190	3157
q14	313	295	283	283
q15	515	486	510	486
q16	493	436	442	436
q17	1815	1535	1525	1525
q18	8033	8035	7810	7810
q19	4853	1548	1631	1548
q20	2091	1907	1835	1835
q21	5467	5251	5368	5251
q22	1100	1048	1032	1032
Total cold run time: 75033 ms
Total hot run time: 55859 ms

doris-robot · 2024-08-12T03:43:30Z

TPC-DS: Total hot run time: 201990 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit e94357a527392964a7364c98d44cb3d0dc236525, data reload: false
query1	1626	1483	1474	1474
query2	6446	2055	1918	1918
query3	13811	8438	303	303
query4	34130	23225	23289	23225
query5	5641	493	482	482
query6	287	168	160	160
query7	3938	307	291	291
query8	267	200	215	200
query9	8443	2485	2466	2466
query10	563	426	456	426
query11	15917	14998	14927	14927
query12	154	95	92	92
query13	1522	356	349	349
query14	10408	6455	7479	6455
query15	293	225	221	221
query16	7366	521	480	480
query17	1503	566	551	551
query18	1937	294	277	277
query19	201	162	142	142
query20	122	107	106	106
query21	206	110	112	110
query22	4450	4505	4296	4296
query23	34248	33590	33701	33590
query24	7351	2543	2486	2486
query25	563	358	366	358
query26	698	152	151	151
query27	2126	283	275	275
query28	5450	2047	2035	2035
query29	781	397	406	397
query30	251	148	147	147
query31	1002	753	737	737
query32	88	52	51	51
query33	494	267	277	267
query34	851	461	466	461
query35	953	843	820	820
query36	1081	930	930	930
query37	130	78	78	78
query38	4216	4172	4150	4150
query39	1420	1394	1367	1367
query40	196	112	112	112
query41	46	43	45	43
query42	121	94	95	94
query43	507	468	462	462
query44	1048	724	718	718
query45	232	212	204	204
query46	1089	736	718	718
query47	1832	1802	1761	1761
query48	368	299	293	293
query49	826	407	435	407
query50	787	399	408	399
query51	6812	6601	6698	6601
query52	107	87	94	87
query53	261	186	216	186
query54	565	440	440	440
query55	75	72	73	72
query56	257	234	240	234
query57	1134	1054	1066	1054
query58	230	227	226	226
query59	3016	2941	2848	2848
query60	294	259	265	259
query61	98	90	104	90
query62	785	617	632	617
query63	206	179	179	179
query64	9146	2250	1763	1763
query65	3208	3129	3137	3129
query66	716	333	320	320
query67	15196	14983	14850	14850
query68	4552	534	529	529
query69	425	379	359	359
query70	1086	1054	1089	1054
query71	392	277	278	277
query72	18090	16737	16665	16665
query73	752	323	324	323
query74	8955	8824	8752	8752
query75	3383	2648	2658	2648
query76	2313	1008	988	988
query77	466	304	300	300
query78	9569	9095	9094	9094
query79	2219	532	530	530
query80	1135	500	500	500
query81	597	218	230	218
query82	796	133	131	131
query83	235	144	142	142
query84	227	77	79	77
query85	1912	276	259	259
query86	444	313	298	298
query87	4675	4468	4511	4468
query88	3850	2493	2497	2493
query89	413	294	281	281
query90	1802	202	196	196
query91	116	95	92	92
query92	61	48	50	48
query93	3231	533	532	532
query94	753	349	280	280
query95	347	261	253	253
query96	603	276	270	270
query97	3228	3054	3051	3051
query98	216	202	193	193
query99	1754	1272	1273	1272
Total cold run time: 308689 ms
Total hot run time: 201990 ms

doris-robot · 2024-08-12T03:49:04Z

ClickBench: Total hot run time: 30.63 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit e94357a527392964a7364c98d44cb3d0dc236525, data reload: false
query1	0.04	0.04	0.04
query2	0.08	0.04	0.04
query3	0.22	0.04	0.05
query4	1.67	0.08	0.08
query5	0.49	0.49	0.50
query6	1.13	0.72	0.72
query7	0.02	0.01	0.01
query8	0.05	0.05	0.05
query9	0.56	0.49	0.49
query10	0.54	0.54	0.54
query11	0.15	0.12	0.12
query12	0.15	0.12	0.12
query13	0.60	0.62	0.59
query14	0.76	0.79	0.80
query15	0.86	0.82	0.82
query16	0.36	0.38	0.39
query17	0.99	1.05	0.99
query18	0.23	0.22	0.21
query19	1.84	1.69	1.67
query20	0.02	0.03	0.01
query21	15.40	0.74	0.66
query22	4.56	7.17	1.82
query23	18.23	1.45	1.29
query24	2.09	0.23	0.21
query25	0.17	0.09	0.08
query26	0.30	0.21	0.22
query27	0.46	0.22	0.23
query28	13.34	1.02	1.02
query29	12.61	3.37	3.31
query30	0.24	0.05	0.05
query31	2.94	0.41	0.39
query32	3.22	0.49	0.49
query33	2.93	2.91	2.91
query34	17.13	4.30	4.35
query35	4.43	4.38	4.45
query36	0.66	0.50	0.47
query37	0.19	0.16	0.16
query38	0.15	0.16	0.16
query39	0.05	0.04	0.04
query40	0.16	0.13	0.12
query41	0.10	0.05	0.05
query42	0.06	0.05	0.05
query43	0.05	0.04	0.04
Total cold run time: 110.23 s
Total hot run time: 30.63 s

When using `JsonFunctions::extract_from_object`, but input obj is not object type but other types like null, then Status should be `simdjson::INCORRECT_TYPE` to fill the default value in the later process`.Example, json path is `$.city.name`, but input json is `{"city" : null}` then `Status::NotFound` should be returned, in the following, column will be filled with default values.

eldenmoon · 2024-08-12T03:56:53Z

run buildall

github-actions · 2024-08-12T04:03:05Z

clang-tidy review says "All clean, LGTM! 👍"

doris-robot · 2024-08-12T04:47:31Z

TPC-H: Total hot run time: 39832 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 272642037c6587f920a7f014738781f71b213cb0, data reload: false
------ Round 1 ----------------------------------
q1	17605	4503	4309	4309
q2	2011	208	200	200
q3	11806	1153	1130	1130
q4	10540	777	836	777
q5	7533	2570	2541	2541
q6	228	140	137	137
q7	971	596	602	596
q8	9415	1930	1941	1930
q9	10242	6623	6610	6610
q10	7021	2257	2224	2224
q11	470	247	242	242
q12	399	227	224	224
q13	17765	3002	3003	3002
q14	281	243	238	238
q15	540	476	489	476
q16	511	403	378	378
q17	977	629	696	629
q18	7959	7364	7479	7364
q19	4509	1134	936	936
q20	679	337	327	327
q21	5393	4569	4562	4562
q22	1155	1000	1031	1000
Total cold run time: 118010 ms
Total hot run time: 39832 ms
----- Round 2, with runtime_filter_mode=off -----
q1	4475	4293	4291	4291
q2	385	260	281	260
q3	2860	2816	2756	2756
q4	2006	1725	1797	1725
q5	5546	5599	5489	5489
q6	227	141	138	138
q7	2224	1830	1815	1815
q8	3316	3458	3461	3458
q9	8790	8889	8811	8811
q10	3570	3225	3198	3198
q11	593	521	508	508
q12	849	676	672	672
q13	17168	3180	3161	3161
q14	314	302	304	302
q15	542	488	494	488
q16	501	455	442	442
q17	1815	1539	1486	1486
q18	8302	8032	7812	7812
q19	6624	1632	1604	1604
q20	2933	1892	1861	1861
q21	9647	5103	5250	5103
q22	1194	1088	1058	1058
Total cold run time: 83881 ms
Total hot run time: 56438 ms

doris-robot · 2024-08-12T05:00:30Z

TPC-DS: Total hot run time: 202288 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 272642037c6587f920a7f014738781f71b213cb0, data reload: false
query1	1612	1464	1454	1454
query2	6334	1905	1852	1852
query3	14265	8923	311	311
query4	33412	23159	23001	23001
query5	3648	476	477	476
query6	254	169	172	169
query7	3933	293	299	293
query8	242	203	202	202
query9	8526	2515	2484	2484
query10	535	499	451	451
query11	17619	15088	14943	14943
query12	141	96	96	96
query13	1530	369	361	361
query14	9729	7874	6349	6349
query15	283	249	254	249
query16	7478	481	486	481
query17	1432	584	584	584
query18	1415	288	296	288
query19	201	145	141	141
query20	114	106	105	105
query21	205	108	102	102
query22	4454	4429	4210	4210
query23	35033	34996	33820	33820
query24	10966	2578	2635	2578
query25	493	363	368	363
query26	697	148	153	148
query27	2121	280	285	280
query28	6203	2071	2063	2063
query29	627	403	417	403
query30	262	147	155	147
query31	993	766	750	750
query32	89	51	53	51
query33	726	281	291	281
query34	860	469	469	469
query35	974	853	848	848
query36	1100	947	942	942
query37	134	79	78	78
query38	4230	4251	4148	4148
query39	1450	1357	1382	1357
query40	211	116	118	116
query41	46	45	46	45
query42	118	103	109	103
query43	493	463	461	461
query44	1201	729	734	729
query45	240	206	206	206
query46	1109	733	721	721
query47	1875	1787	1789	1787
query48	385	308	304	304
query49	833	433	443	433
query50	811	418	412	412
query51	6852	6728	6716	6716
query52	103	94	87	87
query53	273	186	185	185
query54	965	452	454	452
query55	77	76	75	75
query56	272	254	254	254
query57	1152	1102	1100	1100
query58	238	236	237	236
query59	2947	2722	2837	2722
query60	300	268	266	266
query61	117	112	115	112
query62	785	635	661	635
query63	211	185	180	180
query64	9319	2380	1827	1827
query65	3183	3102	3130	3102
query66	713	356	350	350
query67	15645	14830	14913	14830
query68	4739	557	556	556
query69	408	426	416	416
query70	1183	1105	1128	1105
query71	503	281	282	281
query72	19767	16485	16621	16485
query73	762	325	320	320
query74	9024	8856	8934	8856
query75	3538	2714	2697	2697
query76	3618	942	942	942
query77	666	324	307	307
query78	9804	9073	8945	8945
query79	3088	511	514	511
query80	2551	492	494	492
query81	595	223	217	217
query82	793	170	136	136
query83	301	148	150	148
query84	268	82	79	79
query85	1940	287	272	272
query86	468	293	270	270
query87	4786	4474	4683	4474
query88	4282	2517	2491	2491
query89	590	284	293	284
query90	2413	202	195	195
query91	123	94	100	94
query92	65	50	50	50
query93	6374	541	534	534
query94	1197	295	300	295
query95	360	264	260	260
query96	629	278	277	277
query97	3224	3063	3083	3063
query98	216	213	199	199
query99	1609	1263	1276	1263
Total cold run time: 324678 ms
Total hot run time: 202288 ms

doris-robot · 2024-08-12T05:06:04Z

ClickBench: Total hot run time: 30.38 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 272642037c6587f920a7f014738781f71b213cb0, data reload: false
query1	0.05	0.04	0.04
query2	0.08	0.04	0.04
query3	0.22	0.05	0.05
query4	1.66	0.08	0.07
query5	0.49	0.50	0.48
query6	1.12	0.74	0.72
query7	0.02	0.01	0.01
query8	0.06	0.05	0.05
query9	0.56	0.49	0.49
query10	0.56	0.55	0.55
query11	0.15	0.11	0.11
query12	0.15	0.12	0.12
query13	0.60	0.60	0.59
query14	0.76	0.77	0.80
query15	0.88	0.82	0.82
query16	0.36	0.37	0.36
query17	1.04	0.98	0.98
query18	0.22	0.22	0.21
query19	1.82	1.69	1.79
query20	0.02	0.01	0.01
query21	15.39	0.78	0.67
query22	4.53	7.44	1.59
query23	18.23	1.45	1.35
query24	2.09	0.22	0.22
query25	0.16	0.08	0.08
query26	0.29	0.21	0.20
query27	0.45	0.22	0.21
query28	13.33	1.02	1.00
query29	12.59	3.26	3.28
query30	0.24	0.05	0.05
query31	2.89	0.39	0.39
query32	3.25	0.47	0.47
query33	2.94	2.92	2.90
query34	17.04	4.36	4.38
query35	4.42	4.45	4.42
query36	0.65	0.46	0.48
query37	0.20	0.16	0.17
query38	0.16	0.15	0.14
query39	0.05	0.03	0.04
query40	0.15	0.13	0.13
query41	0.10	0.05	0.05
query42	0.06	0.04	0.06
query43	0.05	0.04	0.04
Total cold run time: 110.08 s
Total hot run time: 30.38 s

qidaye

LGTM

github-actions · 2024-08-12T12:12:34Z

PR approved by at least one committer and no changes requested.

amorynan

LGTM

When using `JsonFunctions::extract_from_object`, but input obj is not object type but other types like null, then Status should be `simdjson::INCORRECT_TYPE` to fill the default value in the later process. Example, json path is `$.city.name`, but input json is `{"city" : null}` then `Status::NotFound` should be returned, in the following, column will be filled with default values.

…umn (#39271) #39206 #38213

…39269) #39206

When using `JsonFunctions::extract_from_object`, but input obj is not object type but other types like null, then Status should be `simdjson::INCORRECT_TYPE` to fill the default value in the later process. Example, json path is `$.city.name`, but input json is `{"city" : null}` then `Status::NotFound` should be returned, in the following, column will be filled with default values.

…) (apache#39269) apache#39206

When using `JsonFunctions::extract_from_object`, but input obj is not object type but other types like null, then Status should be `simdjson::INCORRECT_TYPE` to fill the default value in the later process. Example, json path is `$.city.name`, but input json is `{"city" : null}` then `Status::NotFound` should be returned, in the following, column will be filled with default values.

eldenmoon force-pushed the json-load-type branch from 1bdfe13 to e94357aCompare August 12, 2024 02:51

github-actionsBot added the doing label Aug 12, 2024

xiaokang previously approved these changes Aug 12, 2024
View reviewed changes

github-actionsBot added the approved Indicates a PR has been approved by one committer. label Aug 12, 2024

github-actionsBot added the reviewed label Aug 12, 2024

eldenmoon dismissed xiaokang’s stale review via 2726420August 12, 2024 03:56

eldenmoon force-pushed the json-load-type branch from e94357a to 2726420Compare August 12, 2024 03:56

github-actionsBot removed the approved Indicates a PR has been approved by one committer. label Aug 12, 2024

eldenmoon added dev/3.0.x dev/2.0.x dev/2.1.x labels Aug 12, 2024

qidaye approved these changes Aug 12, 2024
View reviewed changes

github-actionsBot added the approved Indicates a PR has been approved by one committer. label Aug 12, 2024

amorynan approved these changes Aug 12, 2024
View reviewed changes

eldenmoon merged commit ee4d0a7 into apache:masterAug 12, 2024

eldenmoon deleted the json-load-type branch August 12, 2024 15:20

eldenmoon mentioned this pull request Aug 13, 2024
[Fix](JsonReader) Return correct status when parse failed (#39206) #39269
Merged

eldenmoon mentioned this pull request Aug 13, 2024
[Pick](Branch-2.1) pick json reader fix and support specify $. as column #39271
Merged

yiguolei pushed a commit that referenced this pull request Aug 13, 2024
[Pick](Branch-2.1) pick json reader fix and support specify $. as col…
677435c
…umn (#39271) #39206 #38213

yiguolei added dev/2.1.6-merged and removed dev/2.1.x labels Aug 13, 2024

eldenmoon added a commit that referenced this pull request Aug 14, 2024
[Fix](JsonReader) Return correct status when parse failed (#39206) (#…
b4d25a8
…39269) #39206

eldenmoon added dev/2.0.15-merged and removed dev/2.0.x labels Aug 14, 2024

dataroaring added dev/3.0.2-merged and removed dev/3.0.x labels Aug 17, 2024

xiaokang removed the doing label Aug 22, 2024

GoGoWen pushed a commit to GoGoWen/incubator-doris that referenced this pull request Aug 27, 2024
[Fix](JsonReader) Return correct status when parse failed (apache#39206…
d437cbb
…) (apache#39269) apache#39206

Uh oh!

Conversation

eldenmoon commented Aug 12, 2024• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Proposed changes

Uh oh!

doris-robot commented Aug 12, 2024

Uh oh!

eldenmoon commented Aug 12, 2024

Uh oh!

eldenmoon commented Aug 12, 2024

Uh oh!

github-actionsBot commented Aug 12, 2024

Uh oh!

github-actionsBot commented Aug 12, 2024

Uh oh!

xiaokang left a comment

Choose a reason for hiding this comment

Uh oh!

github-actionsBot commented Aug 12, 2024

Uh oh!

github-actionsBot commented Aug 12, 2024

Uh oh!

doris-robot commented Aug 12, 2024

Uh oh!

doris-robot commented Aug 12, 2024

Uh oh!

doris-robot commented Aug 12, 2024

Uh oh!

eldenmoon commented Aug 12, 2024

Uh oh!

github-actionsBot commented Aug 12, 2024

Uh oh!

doris-robot commented Aug 12, 2024

Uh oh!

doris-robot commented Aug 12, 2024

Uh oh!

doris-robot commented Aug 12, 2024

Uh oh!

qidaye left a comment

Choose a reason for hiding this comment

Uh oh!

github-actionsBot commented Aug 12, 2024

Uh oh!

amorynan left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

eldenmoon commented Aug 12, 2024•
edited
Loading