Skip to content

[opt](fucntion) improve json_extract function handle const column - #36927

Merged
zhangstar333 merged 5 commits into
apache:masterfrom
zhangstar333:function_json
Jul 3, 2024
Merged

[opt](fucntion) improve json_extract function handle const column#36927
zhangstar333 merged 5 commits into
apache:masterfrom
zhangstar333:function_json

Conversation

@zhangstar333

@zhangstar333zhangstar333 commented Jun 27, 2024•
edited
Loading

Copy link
Copy Markdown
Contributor

Proposed changes

VARCHAR json_extract(VARCHAR json_str, VARCHAR path[, VARCHAR path] ...)

 for (int i = 0; i < arguments.size(); i++) {
column_ptrs.push_back(
block.get_by_position(arguments[i]).column->convert_to_full_column_if_const());
data_columns.push_back(assert_cast<const ColumnString*>(column_ptrs.back().get()));
}
before not handle const column, as the input arguments are variadic,
and most user case is like: json_extract(column, '$.fparam.nested_2')
so could special deal with two arguments could reuse json document.

mysql [test]>select count(json_extract(a, '$.fparam.nested_2')) from json_table_2;
+---------------------------------------------+
| count(json_extract(a, '$.fparam.nested_2')) |
+---------------------------------------------+
| 10000001 |
+---------------------------------------------+
1 row in set (1.06 sec)
mysql [test]>select count(json_extract(a, '$.fparam.nested_2')) from json_table_2;
+---------------------------------------------+
| count(json_extract(a, '$.fparam.nested_2')) |
+---------------------------------------------+
| 10000001 |
+---------------------------------------------+
1 row in set (1.02 sec)
mysql [test]>
mysql [test]>
mysql [test]>select count(json_extract(a, '$.fparam.nested_2')) from json_table_2;
+---------------------------------------------+
| count(json_extract(a, '$.fparam.nested_2')) |
+---------------------------------------------+
| 10000001 |
+---------------------------------------------+
1 row in set (44.22 sec)
mysql [test]>select count(json_extract(a, '$.fparam.nested_2')) from json_table_2;
+---------------------------------------------+
| count(json_extract(a, '$.fparam.nested_2')) |
+---------------------------------------------+
| 10000001 |
+---------------------------------------------+
1 row in set (42.80 sec)

@doris-robot

doris-robot commented Jun 27, 2024

Copy link
Copy Markdown

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@zhangstar333

zhangstar333 commented Jun 27, 2024

Copy link
Copy Markdown
ContributorAuthor

run buildall

@github-actions

github-actionsBot commented Jun 27, 2024

Copy link
Copy Markdown
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@doris-robot

doris-robot commented Jun 27, 2024

Copy link
Copy Markdown
TPC-H: Total hot run time: 40506 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 428419417ec634d0f0f6f2fd7faa13203aeb9220, data reload: false
------ Round 1 ----------------------------------
q1	18126	4875	4296	4296
q2	2021	188	193	188
q3	10567	1187	1075	1075
q4	10228	843	845	843
q5	7501	2782	2613	2613
q6	218	134	133	133
q7	965	599	610	599
q8	9232	2120	2096	2096
q9	8776	6531	6496	6496
q10	8841	3743	3725	3725
q11	471	240	245	240
q12	394	234	224	224
q13	18468	3003	2976	2976
q14	271	226	221	221
q15	511	479	492	479
q16	525	375	382	375
q17	993	618	701	618
q18	8263	7532	7353	7353
q19	6768	1450	1459	1450
q20	662	323	328	323
q21	4987	3841	4051	3841
q22	422	345	342	342
Total cold run time: 119210 ms
Total hot run time: 40506 ms
----- Round 2, with runtime_filter_mode=off -----
q1	4421	4248	4268	4248
q2	368	266	273	266
q3	3185	2940	2928	2928
q4	2010	1697	1754	1697
q5	5502	5504	5501	5501
q6	227	133	137	133
q7	2289	1857	1801	1801
q8	3302	3433	3445	3433
q9	8691	8739	8693	8693
q10	4177	3676	3803	3676
q11	610	494	519	494
q12	821	650	625	625
q13	17146	3095	3127	3095
q14	311	282	275	275
q15	535	489	479	479
q16	511	464	421	421
q17	1854	1536	1517	1517
q18	8033	8151	7901	7901
q19	1796	1627	1632	1627
q20	2086	1875	1859	1859
q21	5071	4944	4871	4871
q22	637	549	548	548
Total cold run time: 73583 ms
Total hot run time: 56088 ms

@doris-robot

doris-robot commented Jun 27, 2024

Copy link
Copy Markdown
TPC-DS: Total hot run time: 174690 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 428419417ec634d0f0f6f2fd7faa13203aeb9220, data reload: false
query1	926	402	376	376
query2	6445	2500	2456	2456
query3	6641	219	221	219
query4	18944	17617	17216	17216
query5	3650	477	464	464
query6	253	189	174	174
query7	4602	297	294	294
query8	307	283	284	283
query9	8498	2366	2335	2335
query10	580	304	274	274
query11	10436	10097	10145	10097
query12	116	88	83	83
query13	1654	373	367	367
query14	10148	7721	7086	7086
query15	274	183	200	183
query16	8090	292	276	276
query17	1870	607	536	536
query18	2120	284	293	284
query19	196	163	159	159
query20	92	85	83	83
query21	216	132	125	125
query22	4494	4373	4244	4244
query23	34085	33519	33644	33519
query24	11070	2931	2948	2931
query25	630	390	374	374
query26	1344	161	154	154
query27	2924	311	326	311
query28	7695	2106	2102	2102
query29	924	648	627	627
query30	253	162	165	162
query31	965	755	751	751
query32	95	54	53	53
query33	746	284	277	277
query34	1050	482	488	482
query35	765	640	638	638
query36	1136	1009	984	984
query37	149	70	76	70
query38	2961	2813	2869	2813
query39	878	844	823	823
query40	223	127	130	127
query41	55	53	51	51
query42	112	98	103	98
query43	577	552	518	518
query44	1200	751	735	735
query45	188	164	167	164
query46	1081	716	706	706
query47	1869	1800	1801	1800
query48	388	307	291	291
query49	865	420	431	420
query50	780	387	393	387
query51	6820	6792	6701	6701
query52	108	91	95	91
query53	352	283	286	283
query54	869	442	437	437
query55	73	76	73	73
query56	318	271	263	263
query57	1134	1097	1061	1061
query58	261	255	235	235
query59	3484	3209	3175	3175
query60	298	273	274	273
query61	96	95	94	94
query62	602	440	450	440
query63	316	282	289	282
query64	9054	2266	1731	1731
query65	3196	3272	3110	3110
query66	1255	327	323	323
query67	15824	15317	15091	15091
query68	5293	541	522	522
query69	661	443	405	405
query70	1179	1140	1105	1105
query71	444	274	273	273
query72	8188	5725	5608	5608
query73	775	326	323	323
query74	5963	5507	5519	5507
query75	4467	2654	2683	2654
query76	3749	966	880	880
query77	666	295	291	291
query78	10502	9816	9789	9789
query79	3191	505	508	505
query80	1439	477	471	471
query81	560	221	228	221
query82	1448	107	106	106
query83	437	179	173	173
query84	325	89	95	89
query85	1524	292	276	276
query86	458	324	300	300
query87	3274	3084	3055	3055
query88	4188	2451	2438	2438
query89	482	381	390	381
query90	1844	187	190	187
query91	126	101	99	99
query92	61	48	51	48
query93	3950	498	496	496
query94	1204	190	183	183
query95	406	311	330	311
query96	599	270	270	270
query97	3282	3083	3040	3040
query98	221	206	198	198
query99	1329	848	841	841
Total cold run time: 281855 ms
Total hot run time: 174690 ms

@doris-robot

doris-robot commented Jun 27, 2024

Copy link
Copy Markdown
ClickBench: Total hot run time: 31.28 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 428419417ec634d0f0f6f2fd7faa13203aeb9220, data reload: false
query1	0.04	0.03	0.03
query2	0.08	0.04	0.04
query3	0.22	0.05	0.05
query4	1.67	0.10	0.08
query5	0.51	0.49	0.51
query6	1.13	0.73	0.72
query7	0.02	0.02	0.01
query8	0.05	0.05	0.05
query9	0.55	0.50	0.50
query10	0.57	0.55	0.56
query11	0.16	0.12	0.11
query12	0.15	0.12	0.12
query13	0.59	0.60	0.61
query14	0.77	0.79	0.76
query15	0.82	0.83	0.82
query16	0.36	0.38	0.37
query17	1.01	1.03	1.03
query18	0.24	0.23	0.26
query19	1.86	1.79	1.71
query20	0.01	0.01	0.01
query21	15.43	0.72	0.64
query22	4.59	6.71	2.51
query23	18.32	1.33	1.20
query24	2.08	0.26	0.21
query25	0.15	0.08	0.09
query26	0.27	0.18	0.17
query27	0.08	0.08	0.08
query28	13.20	1.01	0.99
query29	12.59	3.34	3.31
query30	0.25	0.05	0.05
query31	2.86	0.40	0.38
query32	3.28	0.48	0.48
query33	2.86	2.96	2.90
query34	17.20	4.45	4.42
query35	4.52	4.56	4.53
query36	0.65	0.46	0.50
query37	0.18	0.15	0.16
query38	0.14	0.15	0.15
query39	0.04	0.03	0.04
query40	0.18	0.13	0.14
query41	0.09	0.05	0.05
query42	0.06	0.04	0.05
query43	0.04	0.04	0.04
Total cold run time: 109.87 s
Total hot run time: 31.28 s

@zhangstar333

zhangstar333 commented Jun 27, 2024

Copy link
Copy Markdown
ContributorAuthor

run buildall

@github-actions

github-actionsBot commented Jun 27, 2024

Copy link
Copy Markdown
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@doris-robot

doris-robot commented Jun 27, 2024

Copy link
Copy Markdown
TPC-H: Total hot run time: 39882 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 02a3f4f65dbf3de24bf38b0d600bbd9772613adf, data reload: false
------ Round 1 ----------------------------------
q1	18032	4431	4383	4383
q2	2364	192	191	191
q3	10484	1159	1160	1159
q4	10188	833	835	833
q5	7465	2645	2585	2585
q6	216	137	135	135
q7	934	595	607	595
q8	9230	2058	2059	2058
q9	8880	6461	6498	6461
q10	8862	3731	3729	3729
q11	453	238	235	235
q12	398	234	228	228
q13	17771	3005	3012	3005
q14	255	216	215	215
q15	529	494	487	487
q16	507	369	371	369
q17	962	648	708	648
q18	8142	7481	7284	7284
q19	6205	1401	1524	1401
q20	681	328	343	328
q21	4810	3824	3209	3209
q22	388	344	347	344
Total cold run time: 117756 ms
Total hot run time: 39882 ms
----- Round 2, with runtime_filter_mode=off -----
q1	4364	4221	4217	4217
q2	385	284	259	259
q3	2994	2774	2721	2721
q4	1846	1570	1608	1570
q5	5243	5216	5245	5216
q6	214	128	128	128
q7	2126	1731	1751	1731
q8	3217	3310	3326	3310
q9	8334	8293	8330	8293
q10	3851	3698	3618	3618
q11	582	488	478	478
q12	776	624	602	602
q13	17479	2995	2971	2971
q14	309	251	258	251
q15	528	467	474	467
q16	476	416	462	416
q17	1791	1497	1451	1451
q18	7681	7477	7455	7455
q19	1657	1691	1625	1625
q20	1970	1821	1803	1803
q21	10371	4682	4758	4682
q22	632	576	555	555
Total cold run time: 76826 ms
Total hot run time: 53819 ms

@doris-robot

doris-robot commented Jun 27, 2024

Copy link
Copy Markdown
TPC-DS: Total hot run time: 173377 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 02a3f4f65dbf3de24bf38b0d600bbd9772613adf, data reload: false
query1	917	381	370	370
query2	6469	2384	2378	2378
query3	6655	206	218	206
query4	18843	17625	17227	17227
query5	4198	497	463	463
query6	265	160	162	160
query7	4601	299	292	292
query8	316	298	276	276
query9	8439	2413	2401	2401
query10	613	305	270	270
query11	10589	10128	9962	9962
query12	134	89	84	84
query13	1640	366	360	360
query14	10186	7623	7712	7623
query15	237	186	184	184
query16	8061	287	270	270
query17	1891	541	550	541
query18	2060	270	265	265
query19	195	150	152	150
query20	89	82	81	81
query21	216	131	128	128
query22	4643	4131	4284	4131
query23	33615	33325	32923	32923
query24	11105	2796	2889	2796
query25	621	356	362	356
query26	1389	151	155	151
query27	2988	310	314	310
query28	7554	2086	2079	2079
query29	910	631	622	622
query30	288	158	153	153
query31	956	720	722	720
query32	95	51	59	51
query33	794	309	276	276
query34	969	465	474	465
query35	733	621	613	613
query36	1085	910	940	910
query37	153	71	70	70
query38	2907	2727	2748	2727
query39	859	793	806	793
query40	207	124	124	124
query41	56	51	46	46
query42	116	98	100	98
query43	588	557	572	557
query44	1249	737	728	728
query45	200	168	160	160
query46	1067	735	718	718
query47	1843	1769	1769	1769
query48	367	305	296	296
query49	1082	409	415	409
query50	776	391	402	391
query51	6883	6699	6754	6699
query52	105	91	96	91
query53	362	300	326	300
query54	882	439	455	439
query55	74	72	74	72
query56	277	250	265	250
query57	1108	1055	1040	1040
query58	258	262	244	244
query59	3328	3306	3120	3120
query60	307	274	268	268
query61	94	88	96	88
query62	666	447	462	447
query63	318	292	292	292
query64	9025	2226	1727	1727
query65	3221	3075	3104	3075
query66	1397	349	330	330
query67	15630	15291	14911	14911
query68	4573	558	560	558
query69	466	311	308	308
query70	1106	1154	1126	1126
query71	406	276	273	273
query72	7270	5402	5610	5402
query73	749	323	324	323
query74	5907	5462	5470	5462
query75	3490	2729	2712	2712
query76	2614	945	922	922
query77	432	305	299	299
query78	10416	9857	9750	9750
query79	2282	526	512	512
query80	924	476	465	465
query81	594	221	221	221
query82	771	104	105	104
query83	265	179	165	165
query84	239	87	85	85
query85	1898	281	272	272
query86	480	338	334	334
query87	3306	3065	3100	3065
query88	3914	2379	2372	2372
query89	477	386	375	375
query90	1807	191	192	191
query91	126	103	99	99
query92	66	51	53	51
query93	2435	520	517	517
query94	1334	185	183	183
query95	403	316	316	316
query96	591	271	267	267
query97	3270	3086	3079	3079
query98	213	202	202	202
query99	1141	863	841	841
Total cold run time: 273729 ms
Total hot run time: 173377 ms

@doris-robot

doris-robot commented Jun 27, 2024

Copy link
Copy Markdown
ClickBench: Total hot run time: 30.65 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 02a3f4f65dbf3de24bf38b0d600bbd9772613adf, data reload: false
query1	0.04	0.03	0.03
query2	0.08	0.04	0.04
query3	0.22	0.05	0.05
query4	1.66	0.09	0.07
query5	0.48	0.48	0.48
query6	1.15	0.72	0.72
query7	0.02	0.02	0.01
query8	0.05	0.04	0.04
query9	0.54	0.50	0.50
query10	0.54	0.54	0.53
query11	0.15	0.11	0.12
query12	0.15	0.12	0.13
query13	0.59	0.58	0.60
query14	0.75	0.79	0.78
query15	0.83	0.82	0.81
query16	0.37	0.37	0.35
query17	0.96	1.04	0.96
query18	0.22	0.25	0.21
query19	1.78	1.67	1.70
query20	0.01	0.02	0.01
query21	15.45	0.75	0.66
query22	4.44	7.33	2.21
query23	18.32	1.41	1.23
query24	2.26	0.22	0.22
query25	0.16	0.09	0.09
query26	0.26	0.17	0.18
query27	0.08	0.08	0.09
query28	13.19	1.00	0.99
query29	12.62	3.29	3.25
query30	0.25	0.06	0.05
query31	2.88	0.41	0.39
query32	3.23	0.48	0.48
query33	2.89	2.90	2.91
query34	17.16	4.38	4.39
query35	4.47	4.46	4.50
query36	0.65	0.45	0.45
query37	0.18	0.15	0.15
query38	0.15	0.14	0.13
query39	0.05	0.03	0.04
query40	0.17	0.14	0.14
query41	0.09	0.05	0.05
query42	0.06	0.05	0.04
query43	0.05	0.04	0.03
Total cold run time: 109.65 s
Total hot run time: 30.65 s

@zhangstar333zhangstar333 changed the title [test](fucntion)[opt](fucntion) improve json_extract function handle const columnJun 28, 2024
@zhangstar333

zhangstar333 commented Jun 28, 2024

Copy link
Copy Markdown
ContributorAuthor

run buildall

@github-actions

github-actionsBot commented Jun 28, 2024

Copy link
Copy Markdown
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@doris-robot

doris-robot commented Jun 28, 2024

Copy link
Copy Markdown
TPC-H: Total hot run time: 40229 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 85ba0884ab187259591c172f999ed2faba4d4391, data reload: false
------ Round 1 ----------------------------------
q1	18458	4932	4443	4443
q2	2668	194	189	189
q3	11687	1181	1098	1098
q4	10455	812	731	731
q5	7690	2739	2698	2698
q6	227	138	139	138
q7	971	611	621	611
q8	9390	2142	2092	2092
q9	9048	6556	6515	6515
q10	8817	3696	3735	3696
q11	470	239	237	237
q12	392	239	224	224
q13	18717	2977	3014	2977
q14	260	222	224	222
q15	521	481	481	481
q16	501	371	377	371
q17	994	637	710	637
q18	7927	7434	7421	7421
q19	6050	1514	1481	1481
q20	658	337	341	337
q21	4933	3296	3852	3296
q22	394	334	338	334
Total cold run time: 121228 ms
Total hot run time: 40229 ms
----- Round 2, with runtime_filter_mode=off -----
q1	4407	4224	4267	4224
q2	372	276	261	261
q3	3005	2759	2671	2671
q4	1920	1629	1676	1629
q5	5282	5299	5302	5299
q6	214	128	130	128
q7	2129	1746	1677	1677
q8	3214	3352	3334	3334
q9	8335	8316	8325	8316
q10	3871	3695	3674	3674
q11	594	485	481	481
q12	773	599	676	599
q13	16536	3018	2988	2988
q14	279	266	267	266
q15	523	485	484	484
q16	478	416	432	416
q17	1787	1500	1481	1481
q18	7659	7663	7273	7273
q19	1678	1599	1607	1599
q20	2005	1777	1776	1776
q21	4868	4810	4689	4689
q22	605	534	511	511
Total cold run time: 70534 ms
Total hot run time: 53776 ms

@doris-robot

doris-robot commented Jun 28, 2024

Copy link
Copy Markdown
TPC-DS: Total hot run time: 172842 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 85ba0884ab187259591c172f999ed2faba4d4391, data reload: false
query1	918	373	370	370
query2	6454	2449	2242	2242
query3	6651	204	218	204
query4	19285	17552	17288	17288
query5	4288	479	470	470
query6	267	169	164	164
query7	4624	310	289	289
query8	320	283	283	283
query9	8598	2458	2447	2447
query10	610	295	291	291
query11	10509	10134	10011	10011
query12	130	88	85	85
query13	1660	367	368	367
query14	9231	7728	6914	6914
query15	233	185	197	185
query16	7874	283	271	271
query17	1871	550	528	528
query18	1988	280	276	276
query19	206	152	155	152
query20	90	83	89	83
query21	220	138	128	128
query22	4349	4048	3966	3966
query23	33527	33265	33028	33028
query24	11785	2811	2867	2811
query25	696	383	384	383
query26	1778	155	152	152
query27	2962	315	312	312
query28	7682	2113	2107	2107
query29	1076	640	622	622
query30	297	151	152	151
query31	964	755	758	755
query32	94	53	54	53
query33	789	303	297	297
query34	914	468	481	468
query35	754	633	609	609
query36	1084	954	916	916
query37	162	73	74	73
query38	2865	2755	2775	2755
query39	865	805	775	775
query40	278	128	124	124
query41	59	54	57	54
query42	133	105	105	105
query43	592	542	543	542
query44	1238	748	740	740
query45	204	166	167	166
query46	1081	704	706	704
query47	1889	1778	1767	1767
query48	384	304	298	298
query49	1167	420	422	420
query50	765	379	382	379
query51	6846	6733	6715	6715
query52	107	95	95	95
query53	366	293	295	293
query54	897	469	450	450
query55	77	74	74	74
query56	305	279	293	279
query57	1175	1040	1069	1040
query58	279	260	280	260
query59	3308	3262	2999	2999
query60	301	277	276	276
query61	109	90	103	90
query62	648	466	458	458
query63	317	293	284	284
query64	9830	2245	1772	1772
query65	3154	3121	3075	3075
query66	1401	345	328	328
query67	15558	14989	14985	14985
query68	9151	573	555	555
query69	710	536	391	391
query70	1402	1065	1162	1065
query71	551	272	270	270
query72	9076	5489	5637	5489
query73	2237	327	315	315
query74	5923	5526	5489	5489
query75	5028	2648	2672	2648
query76	5137	1001	831	831
query77	767	311	336	311
query78	10395	9996	9771	9771
query79	7631	517	516	516
query80	997	476	485	476
query81	554	221	217	217
query82	751	104	101	101
query83	358	172	165	165
query84	268	86	87	86
query85	1390	273	268	268
query86	400	317	327	317
query87	3303	3132	3112	3112
query88	4419	2362	2361	2361
query89	550	380	381	380
query90	1938	199	190	190
query91	131	102	97	97
query92	62	51	50	50
query93	5780	519	521	519
query94	1190	187	191	187
query95	408	317	318	317
query96	611	263	268	263
query97	3197	3011	3097	3011
query98	223	198	193	193
query99	1189	828	853	828
Total cold run time: 296865 ms
Total hot run time: 172842 ms

@doris-robot

doris-robot commented Jun 28, 2024

Copy link
Copy Markdown
ClickBench: Total hot run time: 30.47 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 85ba0884ab187259591c172f999ed2faba4d4391, data reload: false
query1	0.04	0.03	0.04
query2	0.07	0.04	0.04
query3	0.23	0.04	0.05
query4	1.69	0.09	0.09
query5	0.49	0.49	0.47
query6	1.13	0.73	0.71
query7	0.02	0.02	0.02
query8	0.06	0.04	0.05
query9	0.56	0.49	0.48
query10	0.53	0.54	0.53
query11	0.15	0.11	0.11
query12	0.15	0.12	0.13
query13	0.59	0.59	0.60
query14	0.77	0.80	0.77
query15	0.84	0.81	0.81
query16	0.36	0.36	0.37
query17	1.02	0.99	1.01
query18	0.22	0.25	0.23
query19	1.87	1.70	1.84
query20	0.01	0.00	0.00
query21	15.44	0.75	0.67
query22	3.72	6.89	1.88
query23	18.31	1.47	1.23
query24	2.08	0.23	0.21
query25	0.15	0.08	0.08
query26	0.26	0.18	0.17
query27	0.08	0.07	0.08
query28	13.27	1.01	1.01
query29	12.63	3.30	3.27
query30	0.26	0.06	0.06
query31	2.88	0.39	0.37
query32	3.29	0.47	0.48
query33	2.93	2.88	2.90
query34	17.32	4.43	4.42
query35	4.46	4.52	4.48
query36	0.65	0.47	0.47
query37	0.18	0.15	0.15
query38	0.15	0.14	0.15
query39	0.05	0.04	0.03
query40	0.17	0.15	0.14
query41	0.09	0.05	0.05
query42	0.06	0.05	0.05
query43	0.04	0.05	0.04
Total cold run time: 109.27 s
Total hot run time: 30.47 s

@zhangstar333

zhangstar333 commented Jun 28, 2024

Copy link
Copy Markdown
ContributorAuthor

run buildall

@github-actions

github-actionsBot commented Jun 28, 2024

Copy link
Copy Markdown
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@doris-robot

doris-robot commented Jun 28, 2024

Copy link
Copy Markdown
TPC-H: Total hot run time: 39772 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 94084bf2cc7fe1aa5c22a33c845c5ac73ab79512, data reload: false
------ Round 1 ----------------------------------
q1	17598	4321	4290	4290
q2	2030	197	189	189
q3	10452	1212	1142	1142
q4	10198	741	881	741
q5	7475	2692	2716	2692
q6	221	133	131	131
q7	946	596	606	596
q8	9231	2087	2085	2085
q9	9056	6454	6496	6454
q10	8895	3713	3653	3653
q11	461	232	242	232
q12	433	239	227	227
q13	17768	2958	2946	2946
q14	259	230	218	218
q15	522	466	466	466
q16	524	376	375	375
q17	972	673	703	673
q18	7983	7384	7409	7384
q19	6977	1508	1545	1508
q20	671	318	325	318
q21	4897	3111	3936	3111
q22	400	341	342	341
Total cold run time: 117969 ms
Total hot run time: 39772 ms
----- Round 2, with runtime_filter_mode=off -----
q1	4381	4248	4230	4230
q2	358	254	261	254
q3	3003	2965	2864	2864
q4	1964	1682	1766	1682
q5	5582	5503	5465	5465
q6	230	128	124	124
q7	2214	1888	1869	1869
q8	3267	3396	3417	3396
q9	8702	8746	8797	8746
q10	4179	3762	3778	3762
q11	589	501	500	500
q12	800	656	627	627
q13	15892	3153	3202	3153
q14	295	272	264	264
q15	527	485	475	475
q16	491	454	425	425
q17	1813	1532	1497	1497
q18	8000	7933	7904	7904
q19	2672	1523	1681	1523
q20	2104	1898	1860	1860
q21	5112	4867	4753	4753
q22	623	566	584	566
Total cold run time: 72798 ms
Total hot run time: 55939 ms

@doris-robot

doris-robot commented Jun 28, 2024

Copy link
Copy Markdown
TPC-DS: Total hot run time: 174389 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 94084bf2cc7fe1aa5c22a33c845c5ac73ab79512, data reload: false
query1	927	370	371	370
query2	6453	2440	2336	2336
query3	6646	210	211	210
query4	19116	17713	17489	17489
query5	3665	467	462	462
query6	267	183	170	170
query7	4586	306	288	288
query8	294	295	297	295
query9	8447	2459	2441	2441
query10	566	301	302	301
query11	10580	10165	10058	10058
query12	115	82	82	82
query13	1649	389	373	373
query14	9707	6998	7880	6998
query15	273	188	194	188
query16	8036	256	255	255
query17	1925	553	516	516
query18	2072	274	265	265
query19	194	149	151	149
query20	93	84	79	79
query21	219	136	130	130
query22	4351	4160	4008	4008
query23	33996	33643	33689	33643
query24	10373	2938	2837	2837
query25	575	374	370	370
query26	709	151	163	151
query27	2258	322	326	322
query28	6098	2174	2187	2174
query29	903	610	625	610
query30	238	159	157	157
query31	959	763	746	746
query32	95	57	51	51
query33	660	287	285	285
query34	895	483	488	483
query35	703	644	619	619
query36	1130	998	970	970
query37	146	74	73	73
query38	2936	2870	2825	2825
query39	883	846	819	819
query40	222	132	124	124
query41	57	53	51	51
query42	109	99	110	99
query43	593	540	531	531
query44	1065	765	741	741
query45	192	158	159	158
query46	1068	730	751	730
query47	1849	1764	1764	1764
query48	365	300	300	300
query49	832	397	413	397
query50	769	386	392	386
query51	6789	6722	6759	6722
query52	104	95	96	95
query53	362	299	296	296
query54	884	441	440	440
query55	73	73	73	73
query56	279	256	260	256
query57	1166	1033	1069	1033
query58	235	237	270	237
query59	3393	3288	3292	3288
query60	294	272	278	272
query61	92	94	91	91
query62	582	441	441	441
query63	327	297	297	297
query64	8564	2232	1716	1716
query65	3201	3101	3095	3095
query66	753	320	319	319
query67	15131	14893	14845	14845
query68	4611	536	548	536
query69	540	494	375	375
query70	1137	1114	1083	1083
query71	363	267	273	267
query72	8079	5672	5546	5546
query73	731	324	325	324
query74	5864	5517	5548	5517
query75	3382	2646	2702	2646
query76	2209	930	931	930
query77	663	311	299	299
query78	10187	9880	9804	9804
query79	2271	517	514	514
query80	2352	461	524	461
query81	570	219	220	219
query82	1430	113	104	104
query83	313	165	173	165
query84	259	93	82	82
query85	1328	285	267	267
query86	467	334	325	325
query87	3229	3062	3162	3062
query88	3805	2378	2368	2368
query89	473	402	386	386
query90	1816	192	191	191
query91	131	100	101	100
query92	61	49	52	49
query93	2328	524	532	524
query94	1188	193	186	186
query95	394	317	317	317
query96	587	267	261	261
query97	3237	3090	3048	3048
query98	210	200	197	197
query99	1124	841	857	841
Total cold run time: 268427 ms
Total hot run time: 174389 ms

@doris-robot

doris-robot commented Jun 28, 2024

Copy link
Copy Markdown
ClickBench: Total hot run time: 30.79 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 94084bf2cc7fe1aa5c22a33c845c5ac73ab79512, data reload: false
query1	0.05	0.03	0.03
query2	0.08	0.04	0.04
query3	0.22	0.04	0.05
query4	1.69	0.07	0.07
query5	0.50	0.49	0.50
query6	1.14	0.74	0.71
query7	0.02	0.01	0.02
query8	0.05	0.05	0.04
query9	0.55	0.48	0.50
query10	0.54	0.54	0.55
query11	0.15	0.11	0.11
query12	0.14	0.11	0.12
query13	0.60	0.59	0.60
query14	0.78	0.80	0.78
query15	0.85	0.82	0.82
query16	0.35	0.36	0.37
query17	1.02	0.99	0.96
query18	0.23	0.21	0.27
query19	1.75	1.79	1.78
query20	0.01	0.01	0.00
query21	15.42	0.73	0.66
query22	4.44	6.98	2.12
query23	18.34	1.34	1.24
query24	2.09	0.23	0.22
query25	0.15	0.08	0.09
query26	0.26	0.18	0.18
query27	0.09	0.08	0.08
query28	13.31	1.02	0.99
query29	12.61	3.30	3.29
query30	0.25	0.07	0.05
query31	2.85	0.39	0.38
query32	3.29	0.47	0.47
query33	2.84	2.92	2.98
query34	17.06	4.40	4.42
query35	4.44	4.47	4.44
query36	0.65	0.47	0.47
query37	0.18	0.16	0.16
query38	0.15	0.15	0.15
query39	0.05	0.03	0.04
query40	0.17	0.15	0.15
query41	0.10	0.05	0.05
query42	0.06	0.06	0.05
query43	0.05	0.04	0.04
Total cold run time: 109.57 s
Total hot run time: 30.79 s

HappenLee
HappenLee reviewed Jul 1, 2024
View reviewed changes
Comment threadbe/src/vec/functions/function_json.cpp Outdated
HappenLee
HappenLee reviewed Jul 1, 2024
View reviewed changes
Comment threadbe/src/vec/functions/function_json.cpp Outdated
HappenLee
HappenLee reviewed Jul 1, 2024
View reviewed changes
Comment threadbe/src/vec/functions/function_json.cpp
@github-actions

github-actionsBot commented Jul 1, 2024

Copy link
Copy Markdown
Contributor

clang-tidy review says "All clean, LGTM! 👍"

HappenLee
HappenLee approved these changes Jul 1, 2024
View reviewed changes

@HappenLeeHappenLee left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions

github-actionsBot commented Jul 1, 2024

Copy link
Copy Markdown
Contributor

PR approved by at least one committer and no changes requested.

@github-actionsgithub-actionsBot added approved Indicates a PR has been approved by one committer. reviewed labels Jul 1, 2024
@github-actions

github-actionsBot commented Jul 1, 2024

Copy link
Copy Markdown
Contributor

PR approved by anyone and no changes requested.

@zhangstar333

zhangstar333 commented Jul 1, 2024

Copy link
Copy Markdown
ContributorAuthor

run buildall

@doris-robot

doris-robot commented Jul 1, 2024

Copy link
Copy Markdown
TPC-H: Total hot run time: 39622 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 55cb5d57a9144e5d3c1f7035ecc5ea9b8b997700, data reload: false
------ Round 1 ----------------------------------
q1	17621	4352	4261	4261
q2	2005	186	189	186
q3	10457	1264	1165	1165
q4	10187	859	787	787
q5	7502	2644	2632	2632
q6	220	136	134	134
q7	963	590	592	590
q8	9224	2068	2070	2068
q9	8869	6456	6487	6456
q10	8967	3715	3712	3712
q11	445	229	228	228
q12	435	238	229	229
q13	17782	2987	2966	2966
q14	249	226	217	217
q15	515	480	492	480
q16	500	383	373	373
q17	960	742	651	651
q18	7979	7495	7305	7305
q19	6046	1475	1408	1408
q20	649	325	321	321
q21	5002	3123	3864	3123
q22	390	330	341	330
Total cold run time: 116967 ms
Total hot run time: 39622 ms
----- Round 2, with runtime_filter_mode=off -----
q1	4488	4279	4222	4222
q2	385	275	276	275
q3	2944	2811	2861	2811
q4	2022	1718	1661	1661
q5	5635	5490	5515	5490
q6	225	135	134	134
q7	2224	1843	1886	1843
q8	3299	3378	3420	3378
q9	8737	8662	8830	8662
q10	4130	3912	3749	3749
q11	584	508	502	502
q12	843	673	647	647
q13	16518	3204	3252	3204
q14	313	273	268	268
q15	534	484	479	479
q16	486	435	432	432
q17	1841	1538	1485	1485
q18	8264	8030	7941	7941
q19	1803	1710	1664	1664
q20	2144	1911	1895	1895
q21	7925	4888	4746	4746
q22	661	589	552	552
Total cold run time: 76005 ms
Total hot run time: 56040 ms

@doris-robot

doris-robot commented Jul 1, 2024

Copy link
Copy Markdown
TPC-DS: Total hot run time: 173312 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 55cb5d57a9144e5d3c1f7035ecc5ea9b8b997700, data reload: false
query1	916	401	375	375
query2	6950	2340	2272	2272
query3	7911	223	218	218
query4	19623	17604	17215	17215
query5	3648	474	465	465
query6	256	174	165	165
query7	4602	290	294	290
query8	295	273	260	260
query9	8525	2359	2337	2337
query10	558	293	281	281
query11	10604	10025	10179	10025
query12	117	81	81	81
query13	1639	361	352	352
query14	9775	7084	7687	7084
query15	255	178	179	178
query16	8073	271	263	263
query17	1898	532	561	532
query18	2102	266	265	265
query19	333	145	148	145
query20	90	80	84	80
query21	206	128	127	127
query22	4576	4217	4195	4195
query23	33910	33503	33565	33503
query24	10335	2890	2946	2890
query25	584	376	381	376
query26	693	151	150	150
query27	2149	322	323	322
query28	5942	2107	2101	2101
query29	865	646	612	612
query30	248	161	176	161
query31	960	756	741	741
query32	91	52	52	52
query33	671	288	282	282
query34	877	482	490	482
query35	744	624	631	624
query36	1124	1000	990	990
query37	133	77	74	74
query38	2967	2816	2875	2816
query39	897	831	827	827
query40	220	131	133	131
query41	54	53	56	53
query42	113	103	105	103
query43	589	555	531	531
query44	1075	732	714	714
query45	186	165	156	156
query46	1071	718	718	718
query47	1884	1795	1759	1759
query48	367	298	296	296
query49	848	395	403	395
query50	763	380	385	380
query51	6789	6760	6717	6717
query52	102	95	91	91
query53	351	284	285	284
query54	879	455	432	432
query55	74	76	74	74
query56	278	256	264	256
query57	1131	1058	1070	1058
query58	252	237	248	237
query59	3492	3017	3148	3017
query60	299	291	320	291
query61	89	92	89	89
query62	582	454	443	443
query63	312	286	291	286
query64	8495	2212	1736	1736
query65	3360	3095	3205	3095
query66	746	324	335	324
query67	15424	15531	14933	14933
query68	4480	545	545	545
query69	607	388	312	312
query70	1169	1121	1119	1119
query71	385	281	270	270
query72	6966	5413	5279	5279
query73	735	322	325	322
query74	6039	5487	5520	5487
query75	3353	2684	2616	2616
query76	2198	932	903	903
query77	431	291	295	291
query78	10370	9898	9816	9816
query79	2308	520	512	512
query80	1030	462	462	462
query81	563	222	208	208
query82	825	102	103	102
query83	234	166	165	165
query84	226	87	86	86
query85	1751	278	271	271
query86	499	307	322	307
query87	3300	3060	3146	3060
query88	3717	2348	2405	2348
query89	488	391	376	376
query90	1775	180	177	177
query91	122	96	98	96
query92	54	47	50	47
query93	2545	498	495	495
query94	1128	184	186	184
query95	401	304	305	304
query96	582	270	260	260
query97	3179	3082	3048	3048
query98	205	203	197	197
query99	1277	840	876	840
Total cold run time: 268909 ms
Total hot run time: 173312 ms

@doris-robot

doris-robot commented Jul 1, 2024

Copy link
Copy Markdown
ClickBench: Total hot run time: 31.06 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 55cb5d57a9144e5d3c1f7035ecc5ea9b8b997700, data reload: false
query1	0.04	0.03	0.02
query2	0.08	0.04	0.04
query3	0.23	0.05	0.05
query4	1.68	0.07	0.07
query5	0.51	0.50	0.49
query6	1.13	0.73	0.72
query7	0.02	0.01	0.02
query8	0.05	0.04	0.04
query9	0.56	0.48	0.47
query10	0.54	0.53	0.56
query11	0.14	0.11	0.10
query12	0.14	0.12	0.11
query13	0.59	0.60	0.58
query14	0.80	0.80	0.79
query15	0.85	0.81	0.82
query16	0.36	0.36	0.35
query17	0.98	0.98	1.03
query18	0.23	0.25	0.22
query19	1.82	1.77	1.72
query20	0.01	0.02	0.01
query21	15.44	0.76	0.65
query22	4.31	6.29	2.42
query23	18.29	1.38	1.21
query24	2.10	0.23	0.23
query25	0.16	0.09	0.09
query26	0.27	0.18	0.18
query27	0.08	0.08	0.08
query28	13.25	1.01	1.00
query29	12.68	3.30	3.30
query30	0.25	0.06	0.05
query31	2.86	0.39	0.39
query32	3.32	0.47	0.46
query33	2.89	2.90	2.95
query34	17.25	4.46	4.46
query35	4.55	4.49	4.52
query36	0.66	0.46	0.48
query37	0.20	0.15	0.15
query38	0.16	0.15	0.14
query39	0.04	0.03	0.04
query40	0.19	0.15	0.14
query41	0.09	0.05	0.04
query42	0.05	0.05	0.05
query43	0.05	0.04	0.03
Total cold run time: 109.9 s
Total hot run time: 31.06 s

@zhangstar333
zhangstar333 merged commit e9500b3 into apache:masterJul 3, 2024
dataroaring pushed a commit that referenced this pull request Jul 17, 2024
@zhangstar333@dataroaring
[opt](fucntion) improve json_extract function handle const column (#3…
d161b9a
…6927)
## Proposed changes
VARCHAR json_extract(VARCHAR json_str, VARCHAR path[, VARCHAR path] ...)
```
for (int i = 0; i < arguments.size(); i++) {
column_ptrs.push_back(
block.get_by_position(arguments[i]).column->convert_to_full_column_if_const());
data_columns.push_back(assert_cast<const ColumnString*>(column_ptrs.back().get()));
}
before not handle const column, as the input arguments are variadic,
and most user case is like: json_extract(column, '$.fparam.nested_2')
so could special deal with two arguments could reuse json document.
```
```
mysql [test]>select count(json_extract(a, '$.fparam.nested_2')) from json_table_2;
+---------------------------------------------+
| count(json_extract(a, '$.fparam.nested_2')) |
+---------------------------------------------+
| 10000001 |
+---------------------------------------------+
1 row in set (1.06 sec)
mysql [test]>select count(json_extract(a, '$.fparam.nested_2')) from json_table_2;
+---------------------------------------------+
| count(json_extract(a, '$.fparam.nested_2')) |
+---------------------------------------------+
| 10000001 |
+---------------------------------------------+
1 row in set (1.02 sec)
mysql [test]>
mysql [test]>
mysql [test]>select count(json_extract(a, '$.fparam.nested_2')) from json_table_2;
+---------------------------------------------+
| count(json_extract(a, '$.fparam.nested_2')) |
+---------------------------------------------+
| 10000001 |
+---------------------------------------------+
1 row in set (44.22 sec)
mysql [test]>select count(json_extract(a, '$.fparam.nested_2')) from json_table_2;
+---------------------------------------------+
| count(json_extract(a, '$.fparam.nested_2')) |
+---------------------------------------------+
| 10000001 |
+---------------------------------------------+
1 row in set (42.80 sec)
```
<!--Describe your changes.-->
@gavinchougavinchou mentioned this pull request Aug 19, 2024
Closed
@yiguoleiyiguolei mentioned this pull request Sep 5, 2024
Closed
3 tasks
HappenLee pushed a commit to HappenLee/incubator-doris that referenced this pull request Apr 24, 2026
@zhangstar333@HappenLee
[opt](fucntion) improve json_extract function handle const column (ap…
386f8b0
…ache#36927)
## Proposed changes
VARCHAR json_extract(VARCHAR json_str, VARCHAR path[, VARCHAR path] ...)
```
for (int i = 0; i < arguments.size(); i++) {
column_ptrs.push_back(
block.get_by_position(arguments[i]).column->convert_to_full_column_if_const());
data_columns.push_back(assert_cast<const ColumnString*>(column_ptrs.back().get()));
}
before not handle const column, as the input arguments are variadic,
and most user case is like: json_extract(column, '$.fparam.nested_2')
so could special deal with two arguments could reuse json document.
```
```
mysql [test]>select count(json_extract(a, '$.fparam.nested_2')) from json_table_2;
+---------------------------------------------+
| count(json_extract(a, '$.fparam.nested_2')) |
+---------------------------------------------+
| 10000001 |
+---------------------------------------------+
1 row in set (1.06 sec)
mysql [test]>select count(json_extract(a, '$.fparam.nested_2')) from json_table_2;
+---------------------------------------------+
| count(json_extract(a, '$.fparam.nested_2')) |
+---------------------------------------------+
| 10000001 |
+---------------------------------------------+
1 row in set (1.02 sec)
mysql [test]>
mysql [test]>
mysql [test]>select count(json_extract(a, '$.fparam.nested_2')) from json_table_2;
+---------------------------------------------+
| count(json_extract(a, '$.fparam.nested_2')) |
+---------------------------------------------+
| 10000001 |
+---------------------------------------------+
1 row in set (44.22 sec)
mysql [test]>select count(json_extract(a, '$.fparam.nested_2')) from json_table_2;
+---------------------------------------------+
| count(json_extract(a, '$.fparam.nested_2')) |
+---------------------------------------------+
| 10000001 |
+---------------------------------------------+
1 row in set (42.80 sec)
```
<!--Describe your changes.-->
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@BiteTheDDDDtBiteTheDDDDtBiteTheDDDDt approved these changes
@HappenLeeHappenLeeHappenLee approved these changes

Assignees

No one assigned

Labels

approvedIndicates a PR has been approved by one committer.dev/3.0.1-mergedreviewed

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@zhangstar333@doris-robot@BiteTheDDDDt@HappenLee@dataroaring