[fix](multi-catalog)put java udf to custom lib by wsjz · Pull Request #34990 · apache/doris

wsjz · 2024-05-17T01:06:06Z

Proposed changes

when use java udf , we can put its jars in custom_lib of FE and BE in advance， and then we can create function without file property:

CREATE FUNCTION java_udf_add_one(int) RETURNS int PROPERTIES (
"symbol"="org.apache.doris.udf.AddOne",
"always_nullable"="true",
"type"="JAVA_UDF"
);

and we can use file property for custom jar:

CREATE FUNCTION java_udf_add_one(int) RETURNS int PROPERTIES (
"file"="file:///path/to/java-udf-demo.jar",
"symbol"="org.apache.doris.udf.AddOne",
"always_nullable"="true",
"type"="JAVA_UDF"
);

on FE, we use null location if the 'file' property is not set
on BE, we get jar path if both file path location and checksum are null

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

doris-robot · 2024-05-17T01:06:10Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

wsjz · 2024-05-22T02:01:02Z

run buildall

doris-robot · 2024-05-22T03:41:07Z

TeamCity be ut coverage result:
Function Coverage: 35.68% (9016/25268)
Line Coverage: 27.32% (74571/272935)
Region Coverage: 26.54% (38557/145303)
Branch Coverage: 23.38% (19664/84096)
Coverage Report: http://coverage.selectdb-in.cc/coverage/6eaff89b212727ae4fabe4489f6e3256b576ec26_6eaff89b212727ae4fabe4489f6e3256b576ec26/report/index.html

wsjz · 2024-05-22T05:44:27Z

run buildall

doris-robot · 2024-05-22T06:18:20Z

TeamCity be ut coverage result:
Function Coverage: 35.68% (9016/25269)
Line Coverage: 27.32% (74581/272958)
Region Coverage: 26.54% (38572/145320)
Branch Coverage: 23.39% (19672/84108)
Coverage Report: http://coverage.selectdb-in.cc/coverage/51aac40c70260a5b50a431f178c7977ec8743442_51aac40c70260a5b50a431f178c7977ec8743442/report/index.html

doris-robot · 2024-05-22T06:22:29Z

TPC-H: Total hot run time: 41889 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 51aac40c70260a5b50a431f178c7977ec8743442, data reload: false
------ Round 1 ----------------------------------
q1	17618	4295	4255	4255
q2	2021	193	210	193
q3	10434	1218	1209	1209
q4	10197	771	872	771
q5	7493	2737	2734	2734
q6	215	143	137	137
q7	1032	632	604	604
q8	9227	2134	2096	2096
q9	9282	6699	6647	6647
q10	9925	3863	3950	3863
q11	455	235	257	235
q12	472	238	237	237
q13	17223	3168	3269	3168
q14	277	222	211	211
q15	518	479	466	466
q16	473	385	400	385
q17	987	742	670	670
q18	8318	8012	7695	7695
q19	3835	1571	1509	1509
q20	628	319	331	319
q21	5356	4206	4296	4206
q22	369	291	279	279
Total cold run time: 116355 ms
Total hot run time: 41889 ms
----- Round 2, with runtime_filter_mode=off -----
q1	4539	4385	4408	4385
q2	371	262	275	262
q3	3141	2961	2841	2841
q4	1911	1636	1650	1636
q5	5506	5473	5515	5473
q6	218	132	125	125
q7	2315	1992	1984	1984
q8	3244	3433	3344	3344
q9	8665	8706	8661	8661
q10	3901	3765	3856	3765
q11	591	499	523	499
q12	811	643	610	610
q13	17084	3180	3189	3180
q14	301	285	291	285
q15	529	488	493	488
q16	479	417	416	416
q17	1752	1460	1454	1454
q18	7790	7675	7460	7460
q19	1657	1542	1582	1542
q20	2000	1773	1755	1755
q21	8224	5022	4875	4875
q22	594	511	480	480
Total cold run time: 75623 ms
Total hot run time: 55520 ms

doris-robot · 2024-05-22T06:33:42Z

TPC-DS: Total hot run time: 182528 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 51aac40c70260a5b50a431f178c7977ec8743442, data reload: false
query1	922	398	378	378
query2	6451	2460	2407	2407
query3	6642	211	211	211
query4	23175	21399	21146	21146
query5	4098	411	411	411
query6	260	172	173	172
query7	4591	287	282	282
query8	238	205	188	188
query9	8397	2378	2366	2366
query10	440	258	254	254
query11	14758	14222	14342	14222
query12	133	90	89	89
query13	1643	387	372	372
query14	9739	8197	8544	8197
query15	258	168	175	168
query16	8049	275	259	259
query17	1730	573	555	555
query18	2111	317	267	267
query19	204	146	147	146
query20	94	86	85	85
query21	191	133	127	127
query22	5029	4813	4920	4813
query23	34503	33593	33588	33588
query24	6725	2838	2910	2838
query25	573	362	367	362
query26	698	154	150	150
query27	1996	311	340	311
query28	3566	2044	2023	2023
query29	835	607	610	607
query30	281	179	186	179
query31	1009	758	727	727
query32	91	51	52	51
query33	474	247	243	243
query34	854	496	490	490
query35	761	679	676	676
query36	1079	955	954	954
query37	109	67	68	67
query38	2909	2765	2775	2765
query39	1603	1558	1555	1555
query40	193	124	124	124
query41	45	43	44	43
query42	102	99	94	94
query43	577	566	557	557
query44	1084	719	737	719
query45	266	246	253	246
query46	1057	733	727	727
query47	1970	1912	1906	1906
query48	365	292	299	292
query49	806	394	393	393
query50	763	382	377	377
query51	6886	6805	6751	6751
query52	105	88	92	88
query53	349	288	280	280
query54	534	433	444	433
query55	75	71	75	71
query56	246	214	222	214
query57	1233	1144	1134	1134
query58	212	202	216	202
query59	3487	3178	3358	3178
query60	245	230	233	230
query61	89	85	86	85
query62	586	477	463	463
query63	305	282	281	281
query64	8416	2241	1732	1732
query65	3200	3072	3075	3072
query66	786	338	335	335
query67	15369	15288	15120	15120
query68	4528	536	550	536
query69	488	300	306	300
query70	1190	1138	1116	1116
query71	387	278	271	271
query72	7178	2576	2387	2387
query73	717	331	334	331
query74	6671	6301	6179	6179
query75	3286	2607	2619	2607
query76	2242	991	952	952
query77	430	261	261	261
query78	10593	10137	10150	10137
query79	2354	503	514	503
query80	927	432	419	419
query81	530	244	243	243
query82	808	98	96	96
query83	240	159	163	159
query84	249	84	84	84
query85	1175	269	264	264
query86	465	317	316	316
query87	3314	3118	3121	3118
query88	4148	2324	2376	2324
query89	477	374	381	374
query90	2064	192	187	187
query91	124	97	96	96
query92	57	47	46	46
query93	1879	514	508	508
query94	1124	179	183	179
query95	399	297	310	297
query96	594	271	263	263
query97	3180	3019	3021	3019
query98	237	219	213	213
query99	1163	937	895	895
Total cold run time: 269788 ms
Total hot run time: 182528 ms

doris-robot · 2024-05-22T06:39:19Z

ClickBench: Total hot run time: 30.89 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 51aac40c70260a5b50a431f178c7977ec8743442, data reload: false
query1	0.04	0.04	0.04
query2	0.08	0.04	0.04
query3	0.23	0.05	0.06
query4	1.67	0.09	0.08
query5	0.49	0.50	0.52
query6	1.12	0.72	0.72
query7	0.02	0.01	0.01
query8	0.05	0.04	0.04
query9	0.55	0.49	0.49
query10	0.55	0.55	0.55
query11	0.15	0.12	0.11
query12	0.15	0.13	0.12
query13	0.60	0.59	0.62
query14	0.77	0.78	0.78
query15	0.84	0.81	0.82
query16	0.36	0.37	0.37
query17	1.03	0.95	1.02
query18	0.21	0.23	0.25
query19	1.79	1.68	1.78
query20	0.02	0.01	0.01
query21	15.81	0.66	0.64
query22	4.45	7.20	2.06
query23	18.26	1.37	1.30
query24	1.49	0.29	0.29
query25	0.14	0.08	0.09
query26	0.26	0.16	0.17
query27	0.07	0.08	0.08
query28	13.30	1.02	0.99
query29	13.15	3.36	3.33
query30	0.23	0.05	0.06
query31	2.87	0.38	0.39
query32	3.26	0.47	0.47
query33	2.88	2.90	2.89
query34	17.09	4.43	4.48
query35	4.55	4.50	4.50
query36	0.73	0.49	0.49
query37	0.18	0.15	0.15
query38	0.15	0.14	0.15
query39	0.04	0.04	0.04
query40	0.16	0.14	0.15
query41	0.09	0.04	0.05
query42	0.06	0.05	0.04
query43	0.04	0.03	0.04
Total cold run time: 109.98 s
Total hot run time: 30.89 s

github-actions · 2024-05-22T11:42:18Z

`sh-checker report`

To get the full details, please check in the job output.

shellcheck errors

'shellcheck ' found no issues.

shfmt errors


'shfmt ' returned error 1 finding the following formatting issues:
----------
--- run-regression-test.sh.orig
+++ run-regression-test.sh
@@ -216,7 +216,7 @@
JAVA_OPTS="${JAVA_OPTS} -DstdoutAppenderType=teamcity -Xmx2048m"
fi
-if [[ "${ONLY_COMPILE}" -eq 0 ]] ; then
+if [[ "${ONLY_COMPILE}" -eq 0 ]]; then
"${JAVA}" -DDORIS_HOME="${DORIS_HOME}" \
-DLOG_PATH="${LOG_OUTPUT_FILE}" \
-Dfile.encoding="UTF-8" \
@@ -226,4 +226,3 @@
-cf "${CONFIG_FILE}" \
${REGRESSION_OPTIONS_PREFIX:+${REGRESSION_OPTIONS_PREFIX}} "$@"
fi
-
----------
You can reformat the above files to meet shfmt's requirements by typing:
shfmt -w filename

github-actions · 2024-05-22T11:44:25Z

`sh-checker report`

To get the full details, please check in the job output.

shellcheck errors

'shellcheck ' found no issues.

shfmt errors


'shfmt ' returned error 1 finding the following formatting issues:
----------
--- run-regression-test.sh.orig
+++ run-regression-test.sh
@@ -215,7 +215,7 @@
JAVA_OPTS="${JAVA_OPTS} -DstdoutAppenderType=teamcity -Xmx2048m"
fi
-if [[ "${ONLY_COMPILE}" -eq 0 ]] ; then
+if [[ "${ONLY_COMPILE}" -eq 0 ]]; then
"${JAVA}" -DDORIS_HOME="${DORIS_HOME}" \
-DLOG_PATH="${LOG_OUTPUT_FILE}" \
-Dfile.encoding="UTF-8" \
@@ -225,4 +225,3 @@
-cf "${CONFIG_FILE}" \
${REGRESSION_OPTIONS_PREFIX:+${REGRESSION_OPTIONS_PREFIX}} "$@"
fi
-
----------
You can reformat the above files to meet shfmt's requirements by typing:
shfmt -w filename

hello-stephen · 2024-05-22T12:04:06Z

run buildall

wsjz · 2024-05-22T12:18:56Z

run buildall

doris-robot · 2024-05-22T12:38:30Z

TeamCity be ut coverage result:
Function Coverage: 35.65% (9012/25277)
Line Coverage: 27.29% (74481/272972)
Region Coverage: 26.51% (38544/145376)
Branch Coverage: 23.36% (19655/84134)
Coverage Report: http://coverage.selectdb-in.cc/coverage/9ebc2dc54dc620f58051efa03f53f4c4c7793e1c_9ebc2dc54dc620f58051efa03f53f4c4c7793e1c/report/index.html

doris-robot · 2024-05-22T12:53:54Z

TPC-H: Total hot run time: 41509 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 9ebc2dc54dc620f58051efa03f53f4c4c7793e1c, data reload: false
------ Round 1 ----------------------------------
q1	17607	4766	4249	4249
q2	2027	195	190	190
q3	10457	1184	1186	1184
q4	10206	766	930	766
q5	7493	2754	2612	2612
q6	223	131	139	131
q7	954	597	612	597
q8	9219	2080	2075	2075
q9	9617	6708	6664	6664
q10	10006	3906	3867	3867
q11	446	242	241	241
q12	471	227	216	216
q13	18213	3193	3177	3177
q14	265	210	220	210
q15	514	473	472	472
q16	512	414	384	384
q17	997	708	737	708
q18	8333	7781	7712	7712
q19	6103	1522	1533	1522
q20	628	319	319	319
q21	5078	3942	4012	3942
q22	345	271	286	271
Total cold run time: 119714 ms
Total hot run time: 41509 ms
----- Round 2, with runtime_filter_mode=off -----
q1	4570	4438	4391	4391
q2	384	259	269	259
q3	3155	2962	2877	2877
q4	1851	1595	1563	1563
q5	5457	5503	5508	5503
q6	215	123	125	123
q7	2176	1838	1830	1830
q8	3242	3385	3319	3319
q9	8597	8663	8621	8621
q10	3966	3739	3827	3739
q11	590	491	493	491
q12	803	612	640	612
q13	16800	3134	3173	3134
q14	302	277	280	277
q15	521	488	487	487
q16	508	447	449	447
q17	1825	1492	1485	1485
q18	7692	7359	7416	7359
q19	1637	1581	1539	1539
q20	1977	1795	1780	1780
q21	4933	4755	4771	4755
q22	568	496	500	496
Total cold run time: 71769 ms
Total hot run time: 55087 ms

doris-robot · 2024-05-22T13:04:41Z

TPC-DS: Total hot run time: 171465 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 9ebc2dc54dc620f58051efa03f53f4c4c7793e1c, data reload: false
query1	925	386	374	374
query2	6469	2452	2314	2314
query3	6636	203	203	203
query4	19412	17431	17176	17176
query5	4093	419	432	419
query6	245	156	154	154
query7	4586	304	287	287
query8	244	191	183	183
query9	8536	2379	2381	2379
query10	449	272	274	272
query11	10767	10227	10094	10094
query12	160	89	86	86
query13	1645	360	360	360
query14	10068	6356	7520	6356
query15	223	172	176	172
query16	7627	268	257	257
query17	1744	524	530	524
query18	1842	276	263	263
query19	201	160	160	160
query20	95	84	86	84
query21	191	138	129	129
query22	4203	3992	4041	3992
query23	33562	33120	32937	32937
query24	9772	2804	2765	2765
query25	580	362	368	362
query26	726	158	156	156
query27	2208	311	314	311
query28	5598	2021	2042	2021
query29	850	622	611	611
query30	276	177	170	170
query31	955	788	760	760
query32	96	53	55	53
query33	652	268	270	268
query34	860	490	489	489
query35	688	615	601	601
query36	1042	925	920	920
query37	109	75	70	70
query38	2915	2788	2798	2788
query39	851	832	800	800
query40	199	127	122	122
query41	45	45	43	43
query42	105	96	98	96
query43	592	526	581	526
query44	1101	722	737	722
query45	183	164	160	160
query46	1058	704	694	694
query47	1844	1766	1744	1744
query48	357	296	305	296
query49	861	378	386	378
query50	773	408	394	394
query51	6959	6767	6914	6767
query52	112	90	93	90
query53	364	303	295	295
query54	787	437	428	428
query55	75	73	74	73
query56	265	258	241	241
query57	1102	1094	1024	1024
query58	228	210	224	210
query59	3413	3216	3140	3140
query60	271	275	258	258
query61	87	90	93	90
query62	597	432	467	432
query63	309	290	288	288
query64	8494	2275	1835	1835
query65	3184	3106	3129	3106
query66	770	339	327	327
query67	15162	14897	14776	14776
query68	4541	552	535	535
query69	445	269	264	264
query70	1218	1097	1169	1097
query71	365	273	274	273
query72	7786	5521	5370	5370
query73	709	323	319	319
query74	5997	5611	5589	5589
query75	3333	2614	2619	2614
query76	2232	1043	1018	1018
query77	380	278	271	271
query78	10304	9677	9729	9677
query79	1940	560	515	515
query80	878	442	439	439
query81	538	250	242	242
query82	956	94	95	94
query83	247	177	167	167
query84	250	88	85	85
query85	1460	274	267	267
query86	456	297	290	290
query87	3282	3224	3127	3127
query88	4015	2312	2316	2312
query89	476	404	396	396
query90	2071	189	188	188
query91	126	97	96	96
query92	60	97	50	50
query93	2101	521	496	496
query94	1266	187	189	187
query95	397	310	313	310
query96	589	266	264	264
query97	3171	3039	2977	2977
query98	260	219	218	218
query99	1199	846	843	843
Total cold run time: 264446 ms
Total hot run time: 171465 ms

doris-robot · 2024-05-22T13:10:10Z

ClickBench: Total hot run time: 30.55 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 9ebc2dc54dc620f58051efa03f53f4c4c7793e1c, data reload: false
query1	0.03	0.03	0.03
query2	0.08	0.04	0.04
query3	0.22	0.05	0.05
query4	1.67	0.08	0.06
query5	0.50	0.48	0.50
query6	1.12	0.72	0.72
query7	0.02	0.01	0.01
query8	0.05	0.05	0.04
query9	0.54	0.49	0.49
query10	0.54	0.55	0.54
query11	0.16	0.11	0.12
query12	0.15	0.12	0.12
query13	0.59	0.59	0.60
query14	0.78	0.78	0.77
query15	0.83	0.80	0.81
query16	0.36	0.37	0.37
query17	0.99	0.94	0.93
query18	0.22	0.23	0.22
query19	1.80	1.72	1.77
query20	0.02	0.01	0.01
query21	15.43	0.71	0.70
query22	4.64	6.79	2.08
query23	18.32	1.39	1.23
query24	1.62	0.32	0.20
query25	0.14	0.08	0.08
query26	0.26	0.16	0.16
query27	0.08	0.08	0.07
query28	13.38	1.03	1.00
query29	13.30	3.28	3.25
query30	0.25	0.05	0.05
query31	2.88	0.38	0.38
query32	3.29	0.47	0.47
query33	2.93	2.91	2.90
query34	17.14	4.41	4.37
query35	4.55	4.47	4.68
query36	0.65	0.48	0.46
query37	0.17	0.16	0.15
query38	0.16	0.15	0.15
query39	0.05	0.03	0.04
query40	0.16	0.16	0.14
query41	0.09	0.05	0.04
query42	0.05	0.04	0.05
query43	0.04	0.04	0.03
Total cold run time: 110.25 s
Total hot run time: 30.55 s

hello-stephen · 2024-05-22T14:00:42Z

run p0

wsjz · 2024-05-23T02:05:03Z

run buildall

doris-robot · 2024-05-23T02:34:49Z

TeamCity be ut coverage result:
Function Coverage: 35.66% (9015/25277)
Line Coverage: 27.31% (74542/272988)
Region Coverage: 26.53% (38579/145394)
Branch Coverage: 23.39% (19679/84148)
Coverage Report: http://coverage.selectdb-in.cc/coverage/d102fa318a2ab656a4b26ce7b787da3691b3810a_d102fa318a2ab656a4b26ce7b787da3691b3810a/report/index.html

doris-robot · 2024-05-23T02:39:21Z

TPC-DS: Total hot run time: 170563 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit d102fa318a2ab656a4b26ce7b787da3691b3810a, data reload: false
query1	926	380	374	374
query2	6182	2622	2509	2509
query3	6654	210	225	210
query4	18945	17944	17689	17689
query5	3933	445	418	418
query6	243	164	156	156
query7	4558	301	307	301
query8	236	181	180	180
query9	8537	2439	2413	2413
query10	424	271	248	248
query11	11343	10917	10566	10566
query12	113	87	90	87
query13	1637	366	371	366
query14	9623	6962	7701	6962
query15	284	168	173	168
query16	8334	273	284	273
query17	1998	535	532	532
query18	2270	277	282	277
query19	203	156	165	156
query20	91	87	89	87
query21	187	141	125	125
query22	4627	3967	4055	3967
query23	33754	32652	33315	32652
query24	10987	2860	2800	2800
query25	591	357	374	357
query26	1109	154	153	153
query27	2882	311	312	311
query28	7505	2034	2040	2034
query29	868	630	603	603
query30	262	168	171	168
query31	958	748	743	743
query32	94	50	55	50
query33	751	264	264	264
query34	946	481	483	481
query35	717	641	590	590
query36	1053	914	936	914
query37	134	77	66	66
query38	2907	2825	2782	2782
query39	832	778	793	778
query40	200	122	123	122
query41	43	47	42	42
query42	100	96	98	96
query43	619	590	554	554
query44	1177	719	727	719
query45	172	162	162	162
query46	1084	712	712	712
query47	1831	1755	1746	1746
query48	361	306	297	297
query49	876	368	368	368
query50	745	386	394	386
query51	6897	6875	6677	6677
query52	95	98	88	88
query53	349	284	285	284
query54	850	421	421	421
query55	75	73	72	72
query56	281	238	253	238
query57	1123	1038	1067	1038
query58	225	199	207	199
query59	3623	3403	3609	3403
query60	292	276	268	268
query61	108	104	106	104
query62	649	470	463	463
query63	321	301	284	284
query64	8896	2288	1783	1783
query65	3127	3090	3062	3062
query66	879	335	318	318
query67	15406	14993	15025	14993
query68	4535	540	547	540
query69	458	267	277	267
query70	1187	1166	1158	1158
query71	435	270	300	270
query72	7559	2711	2509	2509
query73	750	328	329	328
query74	6034	5563	5664	5563
query75	3429	2604	2636	2604
query76	2780	1038	958	958
query77	434	270	271	270
query78	10163	9970	9794	9794
query79	2132	514	512	512
query80	964	431	440	431
query81	558	242	243	242
query82	725	96	95	95
query83	267	164	167	164
query84	229	83	83	83
query85	1503	267	265	265
query86	502	315	309	309
query87	3277	3114	3176	3114
query88	4251	2350	2341	2341
query89	530	376	375	375
query90	1973	193	188	188
query91	128	97	99	97
query92	59	50	51	50
query93	2246	494	502	494
query94	1159	190	192	190
query95	403	317	314	314
query96	589	267	263	263
query97	3162	3001	3013	3001
query98	247	210	221	210
query99	1139	855	857	855
Total cold run time: 271909 ms
Total hot run time: 170563 ms

doris-robot · 2024-05-23T02:44:54Z

ClickBench: Total hot run time: 30.07 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit d102fa318a2ab656a4b26ce7b787da3691b3810a, data reload: false
query1	0.04	0.03	0.03
query2	0.08	0.04	0.04
query3	0.23	0.05	0.05
query4	1.67	0.07	0.09
query5	0.50	0.49	0.51
query6	1.13	0.74	0.73
query7	0.02	0.01	0.02
query8	0.04	0.04	0.05
query9	0.55	0.51	0.48
query10	0.55	0.55	0.55
query11	0.16	0.11	0.11
query12	0.15	0.12	0.12
query13	0.59	0.60	0.61
query14	0.76	0.78	0.77
query15	0.83	0.80	0.81
query16	0.38	0.37	0.35
query17	0.97	0.96	1.01
query18	0.23	0.24	0.25
query19	1.81	1.76	1.69
query20	0.02	0.02	0.02
query21	15.73	0.67	0.66
query22	4.86	7.41	1.50
query23	18.24	1.46	1.27
query24	1.48	0.34	0.20
query25	0.14	0.09	0.08
query26	0.25	0.16	0.16
query27	0.08	0.08	0.08
query28	13.49	1.01	1.00
query29	13.83	3.32	3.24
query30	0.24	0.05	0.06
query31	2.87	0.39	0.38
query32	3.28	0.46	0.46
query33	2.85	2.87	2.93
query34	17.16	4.46	4.44
query35	4.48	4.49	4.66
query36	0.64	0.46	0.47
query37	0.17	0.16	0.16
query38	0.14	0.14	0.14
query39	0.05	0.04	0.04
query40	0.16	0.14	0.13
query41	0.10	0.05	0.04
query42	0.05	0.04	0.04
query43	0.03	0.03	0.04
Total cold run time: 111.03 s
Total hot run time: 30.07 s

wsjz · 2024-05-23T03:47:49Z

run buildall

wsjz · 2024-05-23T08:55:08Z

run buildall

doris-robot · 2024-05-23T09:40:18Z

TeamCity be ut coverage result:
Function Coverage: 35.67% (9018/25279)
Line Coverage: 27.32% (74574/272917)
Region Coverage: 26.56% (38601/145340)
Branch Coverage: 23.41% (19692/84112)
Coverage Report: http://coverage.selectdb-in.cc/coverage/524f5004b6c44a6634ab4e732546a5f841721e3e_524f5004b6c44a6634ab4e732546a5f841721e3e/report/index.html

doris-robot · 2024-05-23T09:41:46Z

TPC-H: Total hot run time: 41813 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 524f5004b6c44a6634ab4e732546a5f841721e3e, data reload: false
------ Round 1 ----------------------------------
q1	17605	4378	4246	4246
q2	2021	191	193	191
q3	10476	1283	1152	1152
q4	10185	841	854	841
q5	7472	2702	2711	2702
q6	232	131	140	131
q7	973	614	605	605
q8	9210	2153	2098	2098
q9	10176	6741	6786	6741
q10	9163	3856	3957	3856
q11	452	240	256	240
q12	431	217	226	217
q13	17652	3162	3290	3162
q14	266	199	212	199
q15	512	484	475	475
q16	504	387	397	387
q17	999	731	730	730
q18	8480	7841	7818	7818
q19	4587	1535	1531	1531
q20	634	308	307	307
q21	5171	3907	4015	3907
q22	375	286	277	277
Total cold run time: 117576 ms
Total hot run time: 41813 ms
----- Round 2, with runtime_filter_mode=off -----
q1	4601	4406	4411	4406
q2	386	269	275	269
q3	3169	2954	2803	2803
q4	1876	1558	1576	1558
q5	5478	5528	5459	5459
q6	217	126	129	126
q7	2147	1792	1852	1792
q8	3246	3416	3435	3416
q9	8581	8652	8638	8638
q10	3946	3692	3806	3692
q11	585	478	474	474
q12	755	627	604	604
q13	16124	3140	3183	3140
q14	300	261	266	261
q15	536	481	500	481
q16	482	431	451	431
q17	1825	1496	1482	1482
q18	7792	7538	7417	7417
q19	1665	1542	1602	1542
q20	1988	1788	1760	1760
q21	9205	4687	4757	4687
q22	552	495	502	495
Total cold run time: 75456 ms
Total hot run time: 54933 ms

doris-robot · 2024-05-23T09:52:25Z

TPC-DS: Total hot run time: 171249 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 524f5004b6c44a6634ab4e732546a5f841721e3e, data reload: false
query1	903	385	374	374
query2	6438	2523	2345	2345
query3	6648	201	202	201
query4	19445	17534	17336	17336
query5	4121	421	424	421
query6	243	154	158	154
query7	4591	299	286	286
query8	234	182	183	182
query9	8452	2456	2434	2434
query10	447	287	283	283
query11	10504	9874	10060	9874
query12	127	89	85	85
query13	1667	367	365	365
query14	9243	6666	7553	6666
query15	227	166	171	166
query16	7761	268	262	262
query17	1690	530	504	504
query18	1967	271	273	271
query19	202	152	150	150
query20	112	81	82	81
query21	190	123	129	123
query22	4185	4116	3823	3823
query23	33520	32920	32981	32920
query24	6723	2828	2876	2828
query25	541	373	372	372
query26	705	157	158	157
query27	2035	317	322	317
query28	4042	2107	2082	2082
query29	857	602	602	602
query30	248	170	175	170
query31	958	763	740	740
query32	92	52	55	52
query33	505	268	280	268
query34	857	487	479	479
query35	734	636	582	582
query36	1040	925	929	925
query37	108	71	69	69
query38	2870	2838	2753	2753
query39	847	791	784	784
query40	210	126	124	124
query41	45	44	43	43
query42	105	99	96	96
query43	573	551	555	551
query44	1081	724	734	724
query45	179	163	167	163
query46	1041	716	713	713
query47	1833	1728	1746	1728
query48	365	301	292	292
query49	766	387	389	387
query50	768	394	383	383
query51	6801	6683	6726	6683
query52	104	85	93	85
query53	356	300	291	291
query54	549	420	429	420
query55	73	70	73	70
query56	259	251	240	240
query57	1158	1016	1052	1016
query58	233	200	206	200
query59	3380	3113	3175	3113
query60	278	261	249	249
query61	88	84	83	83
query62	564	461	441	441
query63	304	284	288	284
query64	8506	2205	1711	1711
query65	3144	3071	3143	3071
query66	788	342	323	323
query67	15127	14675	14622	14622
query68	4553	529	527	527
query69	436	272	275	272
query70	1122	1116	1107	1107
query71	384	266	260	260
query72	7489	5518	5583	5518
query73	726	325	329	325
query74	6039	5690	5600	5600
query75	3351	2624	2637	2624
query76	2258	925	965	925
query77	380	266	265	265
query78	10476	9754	9638	9638
query79	2617	536	526	526
query80	1019	434	427	427
query81	512	247	247	247
query82	889	98	94	94
query83	245	165	168	165
query84	247	85	90	85
query85	1070	274	370	274
query86	462	294	308	294
query87	3254	3118	3040	3040
query88	3898	2444	2422	2422
query89	485	390	400	390
query90	2039	190	192	190
query91	122	95	94	94
query92	58	52	49	49
query93	2014	504	498	498
query94	1156	182	185	182
query95	398	308	307	307
query96	582	271	268	268
query97	3190	2985	3008	2985
query98	236	225	218	218
query99	1169	841	846	841
Total cold run time: 257633 ms
Total hot run time: 171249 ms

doris-robot · 2024-05-23T09:57:55Z

ClickBench: Total hot run time: 30.89 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 524f5004b6c44a6634ab4e732546a5f841721e3e, data reload: false
query1	0.05	0.03	0.04
query2	0.07	0.04	0.03
query3	0.23	0.05	0.05
query4	1.68	0.10	0.10
query5	0.50	0.50	0.51
query6	1.12	0.73	0.74
query7	0.02	0.01	0.02
query8	0.05	0.04	0.04
query9	0.55	0.49	0.49
query10	0.53	0.55	0.54
query11	0.16	0.11	0.10
query12	0.14	0.12	0.13
query13	0.59	0.59	0.59
query14	0.80	0.77	0.77
query15	0.82	0.80	0.80
query16	0.36	0.36	0.36
query17	1.01	1.02	1.01
query18	0.22	0.24	0.26
query19	1.88	1.79	1.79
query20	0.02	0.01	0.01
query21	15.43	0.73	0.68
query22	4.33	7.22	2.10
query23	18.25	1.39	1.27
query24	1.99	0.23	0.21
query25	0.15	0.08	0.08
query26	0.27	0.16	0.17
query27	0.07	0.07	0.08
query28	13.31	1.02	0.99
query29	12.75	3.39	3.30
query30	0.24	0.06	0.05
query31	2.87	0.39	0.38
query32	3.29	0.47	0.46
query33	2.90	2.94	2.87
query34	17.28	4.43	4.45
query35	4.50	4.50	4.49
query36	0.68	0.48	0.46
query37	0.19	0.15	0.15
query38	0.16	0.15	0.15
query39	0.04	0.03	0.04
query40	0.17	0.14	0.14
query41	0.08	0.04	0.05
query42	0.05	0.05	0.04
query43	0.04	0.03	0.04
Total cold run time: 109.84 s
Total hot run time: 30.89 s

wsjz · 2024-05-24T01:48:57Z

run buildall

github-actions · 2024-05-24T01:55:11Z

clang-tidy review says "All clean, LGTM! 👍"

doris-robot · 2024-05-24T02:11:58Z

TPC-H: Total hot run time: 39900 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 416ce095e6fdd72f99405cf18825bc3daa170533, data reload: false
------ Round 1 ----------------------------------
q1	17773	4438	4343	4343
q2	2605	191	193	191
q3	11707	1130	1147	1130
q4	10249	792	786	786
q5	7556	2699	2621	2621
q6	218	133	134	133
q7	968	606	609	606
q8	9279	2065	2050	2050
q9	8888	6476	6492	6476
q10	8874	3722	3689	3689
q11	459	244	240	240
q12	448	221	216	216
q13	17765	2964	2955	2955
q14	244	220	220	220
q15	518	461	473	461
q16	480	378	378	378
q17	960	654	663	654
q18	8138	7541	7315	7315
q19	7388	1543	1513	1513
q20	652	308	310	308
q21	4926	3918	3340	3340
q22	343	275	275	275
Total cold run time: 120438 ms
Total hot run time: 39900 ms
----- Round 2, with runtime_filter_mode=off -----
q1	4311	4200	4227	4200
q2	378	258	278	258
q3	2904	2840	2718	2718
q4	1846	1561	1594	1561
q5	5268	5258	5280	5258
q6	206	126	123	123
q7	2079	1683	1715	1683
q8	3195	3304	3285	3285
q9	8270	8366	8244	8244
q10	3880	3674	3731	3674
q11	566	479	499	479
q12	778	620	561	561
q13	16259	2979	2994	2979
q14	306	257	265	257
q15	519	470	481	470
q16	456	425	423	423
q17	1751	1487	1473	1473
q18	7666	7648	7403	7403
q19	1649	1548	1711	1548
q20	1977	1780	1788	1780
q21	4959	4600	4811	4600
q22	580	475	488	475
Total cold run time: 69803 ms
Total hot run time: 53452 ms

doris-robot · 2024-05-24T02:18:40Z

TeamCity be ut coverage result:
Function Coverage: 35.65% (9018/25294)
Line Coverage: 27.31% (74568/273064)
Region Coverage: 26.53% (38591/145448)
Branch Coverage: 23.39% (19686/84152)
Coverage Report: http://coverage.selectdb-in.cc/coverage/416ce095e6fdd72f99405cf18825bc3daa170533_416ce095e6fdd72f99405cf18825bc3daa170533/report/index.html

doris-robot · 2024-05-24T02:22:55Z

TPC-DS: Total hot run time: 171149 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 416ce095e6fdd72f99405cf18825bc3daa170533, data reload: false
query1	919	374	371	371
query2	6476	2483	2297	2297
query3	6656	207	205	205
query4	19903	17480	17307	17307
query5	4140	415	420	415
query6	253	156	151	151
query7	4586	303	292	292
query8	240	183	172	172
query9	8634	2369	2336	2336
query10	463	284	251	251
query11	10921	10257	10278	10257
query12	126	89	91	89
query13	1638	356	357	356
query14	9220	5898	7530	5898
query15	220	173	165	165
query16	7838	271	258	258
query17	1831	520	534	520
query18	1963	281	280	280
query19	206	166	164	164
query20	93	89	86	86
query21	199	132	131	131
query22	4522	4093	4198	4093
query23	33767	33083	32939	32939
query24	12050	2886	2786	2786
query25	664	350	361	350
query26	1868	155	153	153
query27	3135	321	328	321
query28	7276	2009	2017	2009
query29	1145	616	595	595
query30	315	167	174	167
query31	982	769	764	764
query32	93	51	52	51
query33	762	266	257	257
query34	1047	480	459	459
query35	747	610	612	610
query36	1058	946	924	924
query37	285	69	72	69
query38	2962	2814	2780	2780
query39	845	795	784	784
query40	277	126	124	124
query41	47	44	46	44
query42	99	98	97	97
query43	587	549	562	549
query44	1233	724	732	724
query45	180	162	157	157
query46	1063	736	739	736
query47	1843	1776	1828	1776
query48	365	286	294	286
query49	1178	374	376	374
query50	767	386	375	375
query51	6860	6889	6789	6789
query52	101	90	88	88
query53	349	281	279	279
query54	956	422	436	422
query55	74	76	72	72
query56	263	279	239	239
query57	1128	1051	1033	1033
query58	232	211	205	205
query59	3431	3118	3077	3077
query60	282	244	253	244
query61	85	89	85	85
query62	643	454	455	454
query63	308	283	286	283
query64	9804	2294	1729	1729
query65	3199	3060	3112	3060
query66	1378	325	336	325
query67	15074	15150	14785	14785
query68	4487	530	541	530
query69	446	267	267	267
query70	1186	1124	1147	1124
query71	396	261	276	261
query72	7602	5333	5431	5333
query73	718	323	320	320
query74	6089	5709	5660	5660
query75	3390	2599	2627	2599
query76	2549	955	1031	955
query77	389	278	262	262
query78	10312	9725	9793	9725
query79	2095	513	517	513
query80	928	435	425	425
query81	522	255	249	249
query82	653	95	95	95
query83	237	172	173	172
query84	237	85	86	85
query85	1874	269	253	253
query86	516	305	332	305
query87	3316	3139	3111	3111
query88	4145	2315	2313	2313
query89	470	391	384	384
query90	2014	194	192	192
query91	135	107	108	107
query92	63	50	52	50
query93	2291	508	499	499
query94	1245	198	197	197
query95	476	307	306	306
query96	599	265	262	262
query97	3200	3026	2996	2996
query98	234	230	213	213
query99	1142	863	880	863
Total cold run time: 275810 ms
Total hot run time: 171149 ms

doris-robot · 2024-05-24T02:28:25Z

ClickBench: Total hot run time: 29.72 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 416ce095e6fdd72f99405cf18825bc3daa170533, data reload: false
query1	0.04	0.03	0.04
query2	0.09	0.05	0.04
query3	0.23	0.05	0.05
query4	1.67	0.07	0.07
query5	0.50	0.50	0.52
query6	1.12	0.74	0.73
query7	0.02	0.01	0.02
query8	0.05	0.05	0.04
query9	0.53	0.48	0.49
query10	0.55	0.55	0.55
query11	0.16	0.11	0.11
query12	0.16	0.12	0.12
query13	0.60	0.58	0.58
query14	0.80	0.78	0.76
query15	0.81	0.81	0.82
query16	0.35	0.35	0.36
query17	0.98	0.99	1.00
query18	0.21	0.25	0.23
query19	1.77	1.63	1.66
query20	0.02	0.01	0.02
query21	15.47	0.67	0.64
query22	4.68	7.57	1.31
query23	18.27	1.40	1.26
query24	1.93	0.24	0.20
query25	0.14	0.09	0.08
query26	0.26	0.16	0.16
query27	0.08	0.08	0.08
query28	13.30	1.02	1.00
query29	13.16	3.29	3.26
query30	0.25	0.06	0.06
query31	2.86	0.38	0.38
query32	3.28	0.47	0.47
query33	2.94	2.90	2.88
query34	16.84	4.38	4.40
query35	4.54	4.45	4.49
query36	0.65	0.46	0.46
query37	0.18	0.15	0.15
query38	0.15	0.14	0.14
query39	0.05	0.03	0.03
query40	0.18	0.13	0.15
query41	0.09	0.05	0.04
query42	0.05	0.05	0.04
query43	0.04	0.04	0.03
Total cold run time: 110.05 s
Total hot run time: 29.72 s

morningman

LGTM

morningman · 2024-05-21T15:03:32Z


 package org.apache.doris.analysis;

+import okio.Path;


Wrong import

github-actions · 2024-05-24T09:51:02Z

PR approved by at least one committer and no changes requested.

github-actions · 2024-05-24T09:51:04Z

PR approved by anyone and no changes requested.

when use java udf , we can put its jars in custom_lib of FE and BE in advance， and then we can create function without file property: ``` CREATE FUNCTION java_udf_add_one(int) RETURNS int PROPERTIES ( "symbol"="org.apache.doris.udf.AddOne", "always_nullable"="true", "type"="JAVA_UDF" ); ``` and we can use file property for custom jar: ``` CREATE FUNCTION java_udf_add_one(int) RETURNS int PROPERTIES ( "file"="file:///path/to/java-udf-demo.jar", "symbol"="org.apache.doris.udf.AddOne", "always_nullable"="true", "type"="JAVA_UDF" ); ``` on FE, we use null location if the 'file' property is not set on BE, we get jar path if both file path location and checksum are null

update doc for apache/doris#34990

bp #34990

… jars (#35986) see #34990 user class maybe use some tripartite lib, we should put them back of fe jars to avoid conflicts

bp apache#34990

… jars (apache#35986) see apache#34990 user class maybe use some tripartite lib, we should put them back of fe jars to avoid conflicts

when use java udf , we can put its jars in custom_lib of FE and BE in advance， and then we can create function without file property: ``` CREATE FUNCTION java_udf_add_one(int) RETURNS int PROPERTIES ( "symbol"="org.apache.doris.udf.AddOne", "always_nullable"="true", "type"="JAVA_UDF" ); ``` and we can use file property for custom jar: ``` CREATE FUNCTION java_udf_add_one(int) RETURNS int PROPERTIES ( "file"="file:///path/to/java-udf-demo.jar", "symbol"="org.apache.doris.udf.AddOne", "always_nullable"="true", "type"="JAVA_UDF" ); ``` on FE, we use null location if the 'file' property is not set on BE, we get jar path if both file path location and checksum are null

hello-stephen reviewed May 22, 2024
View reviewed changes

Comment threadrun-regression-test.sh

[fix](multi-catalog)put java udf to custom lib
524f500

morningman added the dev/2.1.x label May 23, 2024

morningman approved these changes May 24, 2024
View reviewed changes

github-actionsBot added the approved Indicates a PR has been approved by one committer. label May 24, 2024

BePPPower approved these changes May 24, 2024
View reviewed changes

github-actionsBot added the reviewed label May 24, 2024

morningman merged commit 1e1a223 into apache:masterMay 26, 2024

wsjz mentioned this pull request May 28, 2024
[doc](java-udf)update udf static load in custom_lib apache/doris-website#687
Merged

luzhijing pushed a commit to apache/doris-website that referenced this pull request May 29, 2024
[doc](java-udf)update udf static load in custom_lib (#687)
888b22b
update doc for apache/doris#34990

This was referenced Jun 6, 2024
[fix](multi-catalog)put java udf to custom lib #35983
Closed
[fix](multi-catalog)put java udf to custom lib #35984
Merged
[fix](multi-catalog)the classpath of custom lib should put back of fe jars #35986
Merged

morningman pushed a commit that referenced this pull request Jun 6, 2024
[fix](multi-catalog)put java udf to custom lib (#35984)
c794ea1
bp #34990

morningman added dev/2.1.4-merged and removed dev/2.1.x labels Jun 6, 2024

morningman pushed a commit that referenced this pull request Jun 7, 2024
[fix](multi-catalog)the classpath of custom lib should put back of fe…
923de47
… jars (#35986) see #34990 user class maybe use some tripartite lib, we should put them back of fe jars to avoid conflicts

morningman added the dev/2.0.12-merged label Jun 7, 2024

morningman pushed a commit to morningman/doris that referenced this pull request Jun 19, 2024
[fix](multi-catalog)put java udf to custom lib (apache#35984)
bc3d402
bp apache#34990

wsjz deleted the cus_lib branch June 27, 2024 09:04

dataroaring added the dev/3.0.0-merged label Aug 5, 2024

Uh oh!

Conversation

wsjz commented May 17, 2024• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Proposed changes

Further comments

Uh oh!

doris-robot commented May 17, 2024

Uh oh!

wsjz commented May 22, 2024

Uh oh!

doris-robot commented May 22, 2024

Uh oh!

wsjz commented May 22, 2024

Uh oh!

doris-robot commented May 22, 2024

Uh oh!

doris-robot commented May 22, 2024

Uh oh!

doris-robot commented May 22, 2024

Uh oh!

doris-robot commented May 22, 2024

Uh oh!

github-actionsBot commented May 22, 2024

sh-checker report

Uh oh!

github-actionsBot commented May 22, 2024

sh-checker report

Uh oh!

hello-stephen commented May 22, 2024

Uh oh!

wsjz commented May 22, 2024

Uh oh!

doris-robot commented May 22, 2024

Uh oh!

doris-robot commented May 22, 2024

Uh oh!

doris-robot commented May 22, 2024

Uh oh!

doris-robot commented May 22, 2024

Uh oh!

Uh oh!

hello-stephen commented May 22, 2024

Uh oh!

wsjz commented May 23, 2024

Uh oh!

doris-robot commented May 23, 2024

Uh oh!

doris-robot commented May 23, 2024

Uh oh!

doris-robot commented May 23, 2024

Uh oh!

wsjz commented May 23, 2024

Uh oh!

wsjz commented May 23, 2024

Uh oh!

doris-robot commented May 23, 2024

Uh oh!

doris-robot commented May 23, 2024

Uh oh!

doris-robot commented May 23, 2024

Uh oh!

doris-robot commented May 23, 2024

Uh oh!

wsjz commented May 24, 2024

Uh oh!

github-actionsBot commented May 24, 2024

Uh oh!

doris-robot commented May 24, 2024

Uh oh!

doris-robot commented May 24, 2024

Uh oh!

doris-robot commented May 24, 2024

Uh oh!

doris-robot commented May 24, 2024

Uh oh!

morningman left a comment

Choose a reason for hiding this comment

Uh oh!

morningmanMay 21, 2024

wsjz commented May 17, 2024•
edited
Loading

`sh-checker report`

`sh-checker report`