[feat](jdbc) support custom function rules in catalog properties by morningman · Pull Request #51471 · apache/doris

morningman · 2025-06-04T05:20:20Z

What problem does this PR solve?

Add new jdbc catalog property: function_rules.
This property will enable user to customize which function can be pushed down to jdbc datasource,
and which function should be rewrote to function in jdbc datasource:

eg:

create catalog jdbc properties (
...
'function_rules' = '{"pushdown" : {"supported": ["to_date"], "unsupported" : ["abs"]}, "rewrite" : {"to_date" : "date2"}}'
)

The value is a json format config:

pushdown: the value of pushdown is a map, the supported array indicate the function that can be pushed down to datasource. the unsupported array indicate the function that can not.
rewrite: the value of rewrite is map, if the function can be pushed down to datasource, it will be renamed to the value in rewrite map.

eg:

select * from jdbc_table where abs(k1) > 0 and to_date("k2") = "2022-10-10"

will generate sql:

select * from jdbc_table where date2("k2") = "2022-10-10"

You can see the abs() is not pushed down, and to_date() is rewrote to date2().

This change is compatible with previous version.
Previously, there are some hard coded function rules, and also a FE config jdbc_mysql_unsupported_pushdown_functions.

Doris will merge the user specified function_rules and the above rules.

Release note

None

Check List (For Author)

Test
- Regression test
- Unit Test
- Manual test (add detailed scripts or steps below)
- No need to test or manual test. Explain why:
  - This is a refactor/code format and no logic has been changed.
  - Previous test can cover this change.
  - No code files have been changed.
  - Other reason
Behavior changed:
- No.
- Yes.
Does this need documentation?
- No.
- Yes. [feat](jdbc) add jdbc function rules doris-website#2470

Check List (For Reviewer who merge this PR)

Confirm the release note
Confirm test cases
Confirm document
Add branch pick label

hello-stephen · 2025-06-04T05:20:26Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

What problem was fixed (it's best to include specific error reporting information). How it was fixed.
Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
What features were added. Why was this function added?
Which code was refactored and why was this part of the code refactored?
Which functions were optimized and what is the difference before and after the optimization?

morningman · 2025-06-04T10:22:04Z

run buildall

morningman · 2025-06-04T11:15:34Z

run buildall

doris-robot · 2025-06-04T11:41:16Z

TPC-H: Total hot run time: 33856 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit c3a6206c1ef33ae0f4eb5b8e79e3e372f4efb1bb, data reload: false
------ Round 1 ----------------------------------
q1	26279	5087	5000	5000
q2	1957	272	178	178
q3	10294	1257	686	686
q4	10239	1001	530	530
q5	7532	2422	2303	2303
q6	190	165	138	138
q7	912	741	603	603
q8	9313	1278	1095	1095
q9	6841	5135	5187	5135
q10	6875	2306	1879	1879
q11	497	290	281	281
q12	355	342	219	219
q13	17793	3623	3104	3104
q14	225	227	209	209
q15	568	479	505	479
q16	432	431	366	366
q17	592	861	358	358
q18	7607	7211	7265	7211
q19	1499	928	550	550
q20	326	335	217	217
q21	3732	3192	2360	2360
q22	1052	1029	955	955
Total cold run time: 115110 ms
Total hot run time: 33856 ms
----- Round 2, with runtime_filter_mode=off -----
q1	5160	5027	5066	5027
q2	240	316	225	225
q3	2179	2627	2298	2298
q4	1380	1778	1319	1319
q5	4450	4350	4453	4350
q6	206	172	131	131
q7	2023	1902	1742	1742
q8	2582	2593	2508	2508
q9	7233	7257	6988	6988
q10	3001	3205	2765	2765
q11	590	505	485	485
q12	683	781	601	601
q13	3539	3882	3310	3310
q14	300	293	267	267
q15	535	483	479	479
q16	454	508	428	428
q17	1122	1522	1386	1386
q18	7806	7553	7451	7451
q19	804	794	906	794
q20	1979	2082	1833	1833
q21	4687	4411	4448	4411
q22	1101	1077	1013	1013
Total cold run time: 52054 ms
Total hot run time: 49811 ms

doris-robot · 2025-06-04T11:53:22Z

TPC-DS: Total hot run time: 188523 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit c3a6206c1ef33ae0f4eb5b8e79e3e372f4efb1bb, data reload: false
query1	1400	1078	1055	1055
query2	6310	1776	1799	1776
query3	11009	4562	4182	4182
query4	55298	25665	22915	22915
query5	5069	499	485	485
query6	348	199	190	190
query7	4875	509	291	291
query8	277	225	209	209
query9	5468	2637	2651	2637
query10	450	330	273	273
query11	14931	15033	14836	14836
query12	165	107	107	107
query13	1054	529	421	421
query14	10198	6208	6155	6155
query15	211	205	193	193
query16	7132	649	417	417
query17	1068	733	594	594
query18	1557	408	302	302
query19	202	192	162	162
query20	124	133	117	117
query21	213	130	109	109
query22	4350	4450	4394	4394
query23	34434	33418	33663	33418
query24	6584	2406	2435	2406
query25	461	467	406	406
query26	715	268	153	153
query27	2282	508	348	348
query28	3064	2171	2164	2164
query29	587	552	444	444
query30	271	221	196	196
query31	853	855	791	791
query32	71	63	60	60
query33	470	360	311	311
query34	791	876	537	537
query35	812	870	780	780
query36	931	991	894	894
query37	111	102	77	77
query38	4242	4338	4187	4187
query39	1518	1467	1486	1467
query40	211	122	112	112
query41	61	57	58	57
query42	126	108	110	108
query43	513	507	477	477
query44	1333	843	850	843
query45	181	172	174	172
query46	870	1024	641	641
query47	1795	1918	1776	1776
query48	415	421	329	329
query49	679	490	430	430
query50	652	726	406	406
query51	4178	4310	4191	4191
query52	114	106	98	98
query53	221	260	180	180
query54	581	587	515	515
query55	87	86	87	86
query56	304	344	279	279
query57	1176	1181	1104	1104
query58	280	267	256	256
query59	2600	2789	2637	2637
query60	341	318	321	318
query61	124	127	123	123
query62	764	778	686	686
query63	233	179	185	179
query64	1453	1011	673	673
query65	4268	4132	4142	4132
query66	695	405	312	312
query67	15839	15544	15354	15354
query68	5640	899	545	545
query69	552	305	278	278
query70	1183	1089	1080	1080
query71	451	316	320	316
query72	5992	2336	5076	2336
query73	1359	758	350	350
query74	9225	9134	8633	8633
query75	3405	3191	2709	2709
query76	3741	1185	757	757
query77	550	357	296	296
query78	10201	10075	9259	9259
query79	2210	811	574	574
query80	604	582	432	432
query81	499	253	224	224
query82	452	126	94	94
query83	263	254	235	235
query84	298	98	91	91
query85	812	347	311	311
query86	408	299	281	281
query87	4358	4409	4373	4373
query88	3515	2256	2245	2245
query89	404	316	282	282
query90	1850	205	199	199
query91	139	139	110	110
query92	73	112	56	56
query93	1838	940	587	587
query94	719	407	249	249
query95	363	290	276	276
query96	486	565	281	281
query97	2719	2771	2675	2675
query98	238	213	200	200
query99	1444	1427	1256	1256
Total cold run time: 295991 ms
Total hot run time: 188523 ms

doris-robot · 2025-06-04T11:58:32Z

ClickBench: Total hot run time: 29.46 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit c3a6206c1ef33ae0f4eb5b8e79e3e372f4efb1bb, data reload: false
query1	0.04	0.04	0.03
query2	0.12	0.11	0.10
query3	0.26	0.19	0.19
query4	1.59	0.20	0.19
query5	0.46	0.44	0.46
query6	1.15	0.66	0.66
query7	0.02	0.02	0.01
query8	0.04	0.03	0.03
query9	0.58	0.52	0.50
query10	0.57	0.57	0.56
query11	0.16	0.11	0.11
query12	0.14	0.12	0.11
query13	0.61	0.60	0.59
query14	0.79	0.80	0.80
query15	0.87	0.87	0.86
query16	0.37	0.38	0.38
query17	1.01	1.01	1.02
query18	0.23	0.21	0.21
query19	1.93	1.90	1.82
query20	0.02	0.01	0.02
query21	15.39	0.90	0.56
query22	0.77	1.19	0.71
query23	14.82	1.37	0.62
query24	7.03	1.20	1.26
query25	0.53	0.19	0.08
query26	0.55	0.17	0.15
query27	0.06	0.05	0.04
query28	10.29	0.93	0.44
query29	12.58	3.94	3.28
query30	0.26	0.09	0.07
query31	2.82	0.59	0.39
query32	3.23	0.54	0.47
query33	3.07	3.10	3.11
query34	15.40	5.07	4.46
query35	4.50	4.47	4.51
query36	0.68	0.50	0.49
query37	0.09	0.06	0.06
query38	0.06	0.03	0.04
query39	0.03	0.02	0.03
query40	0.17	0.14	0.13
query41	0.08	0.03	0.03
query42	0.03	0.02	0.03
query43	0.04	0.04	0.03
Total cold run time: 103.44 s
Total hot run time: 29.46 s

morningman · 2025-06-06T02:28:58Z

run buildall

morningman · 2025-06-06T02:41:31Z

run buildall

doris-robot · 2025-06-06T03:31:15Z

TPC-H: Total hot run time: 33963 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit ddc3281c1445b6e4dd842d0e2de3d72832ab3e20, data reload: false
------ Round 1 ----------------------------------
q1	26411	5157	5172	5157
q2	1960	280	187	187
q3	10278	1228	688	688
q4	10221	1037	526	526
q5	7508	2400	2339	2339
q6	188	161	132	132
q7	917	761	623	623
q8	9296	1249	1078	1078
q9	6941	5138	5056	5056
q10	6866	2334	1904	1904
q11	502	290	275	275
q12	342	351	212	212
q13	17784	3766	3098	3098
q14	237	249	210	210
q15	568	486	494	486
q16	424	438	372	372
q17	595	857	356	356
q18	7673	7239	7187	7187
q19	1574	952	549	549
q20	337	347	218	218
q21	3681	3197	2366	2366
q22	1066	1060	944	944
Total cold run time: 115369 ms
Total hot run time: 33963 ms
----- Round 2, with runtime_filter_mode=off -----
q1	5172	5109	5147	5109
q2	247	320	220	220
q3	2163	2645	2309	2309
q4	1378	1813	1396	1396
q5	4488	4404	4437	4404
q6	205	167	125	125
q7	2002	1948	1833	1833
q8	2605	2566	2599	2566
q9	7284	7138	7082	7082
q10	3039	3199	2779	2779
q11	582	506	526	506
q12	743	773	614	614
q13	3538	3895	3284	3284
q14	283	318	285	285
q15	545	490	464	464
q16	426	488	436	436
q17	1133	1543	1382	1382
q18	7924	7679	7511	7511
q19	820	831	846	831
q20	2003	1966	1887	1887
q21	4820	4535	4488	4488
q22	1116	1095	1050	1050
Total cold run time: 52516 ms
Total hot run time: 50561 ms

doris-robot · 2025-06-06T03:43:14Z

TPC-DS: Total hot run time: 193058 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit ddc3281c1445b6e4dd842d0e2de3d72832ab3e20, data reload: false
query1	1397	1089	1075	1075
query2	6461	1870	1839	1839
query3	11147	4656	4841	4656
query4	26143	24098	23385	23385
query5	4812	638	490	490
query6	324	217	200	200
query7	3992	493	285	285
query8	263	223	218	218
query9	8501	2641	2630	2630
query10	500	341	281	281
query11	15476	15005	14825	14825
query12	160	113	113	113
query13	1559	529	430	430
query14	9036	6185	6054	6054
query15	205	194	173	173
query16	7521	637	484	484
query17	1164	794	639	639
query18	2070	426	311	311
query19	221	187	164	164
query20	131	127	125	125
query21	214	131	115	115
query22	4498	4538	4271	4271
query23	35013	33857	33569	33569
query24	8723	2431	2397	2397
query25	519	473	407	407
query26	1247	281	157	157
query27	2717	520	348	348
query28	4800	2199	2199	2199
query29	711	587	446	446
query30	275	230	197	197
query31	954	885	785	785
query32	71	65	61	61
query33	556	374	308	308
query34	835	880	568	568
query35	809	839	749	749
query36	970	987	879	879
query37	115	97	81	81
query38	4339	4334	4145	4145
query39	1546	1508	1475	1475
query40	210	121	114	114
query41	60	62	58	58
query42	125	111	110	110
query43	507	547	481	481
query44	1354	832	833	832
query45	190	182	166	166
query46	847	1027	656	656
query47	1895	1915	1818	1818
query48	389	424	332	332
query49	740	531	376	376
query50	655	700	413	413
query51	4280	4271	4186	4186
query52	122	115	99	99
query53	228	256	194	194
query54	597	569	508	508
query55	81	88	82	82
query56	330	298	287	287
query57	1187	1217	1173	1173
query58	277	266	261	261
query59	2766	2867	2790	2790
query60	350	336	301	301
query61	124	123	121	121
query62	780	754	686	686
query63	231	190	195	190
query64	4162	1046	775	775
query65	4339	4291	4231	4231
query66	1110	429	374	374
query67	16130	15748	15560	15560
query68	9353	903	528	528
query69	488	301	283	283
query70	1204	1100	1128	1100
query71	477	319	300	300
query72	5602	4690	4707	4690
query73	707	603	363	363
query74	9273	9116	8889	8889
query75	4311	3211	2686	2686
query76	3759	1197	771	771
query77	931	362	277	277
query78	10039	10328	9339	9339
query79	3185	833	565	565
query80	681	522	431	431
query81	479	265	217	217
query82	252	122	93	93
query83	281	250	238	238
query84	289	111	95	95
query85	763	373	381	373
query86	327	293	276	276
query87	4390	4467	4218	4218
query88	2888	2299	2286	2286
query89	435	317	289	289
query90	2138	210	204	204
query91	150	139	112	112
query92	80	71	57	57
query93	2241	952	570	570
query94	680	456	310	310
query95	371	291	298	291
query96	491	570	287	287
query97	2730	2772	2715	2715
query98	236	210	203	203
query99	1421	1411	1294	1294
Total cold run time: 284713 ms
Total hot run time: 193058 ms

doris-robot · 2025-06-06T03:48:24Z

ClickBench: Total hot run time: 29.16 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit ddc3281c1445b6e4dd842d0e2de3d72832ab3e20, data reload: false
query1	0.04	0.04	0.03
query2	0.13	0.11	0.11
query3	0.25	0.19	0.20
query4	1.60	0.20	0.20
query5	0.46	0.45	0.45
query6	1.16	0.66	0.66
query7	0.02	0.02	0.02
query8	0.04	0.04	0.03
query9	0.60	0.51	0.52
query10	0.56	0.58	0.58
query11	0.16	0.11	0.11
query12	0.15	0.12	0.12
query13	0.62	0.60	0.62
query14	0.79	0.80	0.82
query15	0.89	0.87	0.88
query16	0.38	0.37	0.38
query17	1.04	1.01	1.07
query18	0.23	0.21	0.21
query19	1.92	1.78	1.91
query20	0.01	0.01	0.02
query21	15.43	0.91	0.54
query22	0.77	1.26	0.74
query23	14.78	1.40	0.61
query24	7.10	1.41	0.63
query25	0.49	0.19	0.16
query26	0.63	0.17	0.15
query27	0.05	0.05	0.05
query28	9.42	0.89	0.46
query29	12.55	4.04	3.35
query30	0.26	0.10	0.07
query31	2.82	0.61	0.39
query32	3.24	0.57	0.47
query33	3.05	3.05	3.04
query34	15.66	5.14	4.52
query35	4.53	4.56	4.53
query36	0.67	0.50	0.48
query37	0.08	0.06	0.06
query38	0.05	0.04	0.03
query39	0.03	0.02	0.02
query40	0.16	0.13	0.13
query41	0.08	0.02	0.02
query42	0.03	0.03	0.03
query43	0.04	0.02	0.03
Total cold run time: 102.97 s
Total hot run time: 29.16 s

morningman · 2025-06-09T06:42:33Z

run buildall

doris-robot · 2025-06-09T07:19:54Z

TPC-H: Total hot run time: 33633 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 34c78f15f5010cc145c15121febca4fe028b6aaf, data reload: false
------ Round 1 ----------------------------------
q1	25998	5220	5181	5181
q2	1933	281	181	181
q3	10299	1225	689	689
q4	10224	985	509	509
q5	7525	2343	2276	2276
q6	172	163	127	127
q7	885	745	611	611
q8	9316	1245	1024	1024
q9	6983	5163	5180	5163
q10	6855	2322	1883	1883
q11	481	286	271	271
q12	339	348	210	210
q13	17773	3635	3050	3050
q14	231	225	210	210
q15	570	477	489	477
q16	429	431	374	374
q17	576	838	360	360
q18	7429	7212	7035	7035
q19	1857	950	522	522
q20	324	327	217	217
q21	3633	2523	2276	2276
q22	1044	1022	987	987
Total cold run time: 114876 ms
Total hot run time: 33633 ms
----- Round 2, with runtime_filter_mode=off -----
q1	5512	5406	5378	5378
q2	239	312	219	219
q3	2165	2619	2313	2313
q4	1410	1861	1401	1401
q5	4519	4391	4540	4391
q6	221	168	127	127
q7	2030	1964	1797	1797
q8	2609	2592	2526	2526
q9	7421	7359	7154	7154
q10	3081	3238	2791	2791
q11	588	521	513	513
q12	684	766	616	616
q13	3527	3897	3264	3264
q14	287	296	282	282
q15	540	478	465	465
q16	457	480	443	443
q17	1137	1520	1393	1393
q18	7979	7724	7388	7388
q19	777	855	1038	855
q20	1980	2011	1838	1838
q21	4899	4500	4498	4498
q22	1086	1064	1047	1047
Total cold run time: 53148 ms
Total hot run time: 50699 ms

doris-robot · 2025-06-09T07:31:49Z

TPC-DS: Total hot run time: 192566 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 34c78f15f5010cc145c15121febca4fe028b6aaf, data reload: false
query1	1430	1103	1057	1057
query2	6165	1803	1791	1791
query3	11151	4807	4708	4708
query4	25828	24064	22985	22985
query5	4759	617	462	462
query6	313	213	206	206
query7	3988	500	282	282
query8	285	228	215	215
query9	8523	2606	2613	2606
query10	466	329	264	264
query11	15414	15056	14738	14738
query12	166	114	106	106
query13	1565	533	413	413
query14	8820	6182	6047	6047
query15	202	208	170	170
query16	7310	628	498	498
query17	1159	698	617	617
query18	1973	413	341	341
query19	231	189	162	162
query20	130	124	134	124
query21	208	133	109	109
query22	4494	4645	4385	4385
query23	34943	33770	33805	33770
query24	8552	2389	2437	2389
query25	525	478	401	401
query26	1214	275	150	150
query27	2793	530	338	338
query28	4614	2189	2144	2144
query29	753	595	467	467
query30	283	228	195	195
query31	945	886	808	808
query32	74	73	72	72
query33	550	377	319	319
query34	810	897	559	559
query35	823	832	754	754
query36	976	1007	879	879
query37	108	96	77	77
query38	4128	4390	4219	4219
query39	1535	1446	1496	1446
query40	224	124	106	106
query41	60	59	58	58
query42	122	109	118	109
query43	510	502	484	484
query44	1347	850	866	850
query45	182	172	173	172
query46	864	1027	656	656
query47	1871	1905	1800	1800
query48	390	416	312	312
query49	735	494	394	394
query50	681	690	405	405
query51	4698	4228	4179	4179
query52	118	112	109	109
query53	235	272	184	184
query54	603	567	524	524
query55	85	85	87	85
query56	311	322	282	282
query57	1213	1227	1197	1197
query58	302	276	275	275
query59	2711	2777	2646	2646
query60	336	316	324	316
query61	126	122	121	121
query62	786	750	664	664
query63	235	198	192	192
query64	4259	1131	671	671
query65	4327	4246	4245	4245
query66	1035	403	298	298
query67	15855	15773	15511	15511
query68	8834	887	523	523
query69	486	298	275	275
query70	1190	1080	1118	1080
query71	508	324	286	286
query72	5612	4717	4625	4625
query73	695	567	356	356
query74	9388	9050	9162	9050
query75	3990	3201	2703	2703
query76	3754	1185	755	755
query77	803	373	282	282
query78	10112	10149	9365	9365
query79	2727	791	569	569
query80	803	504	450	450
query81	491	253	222	222
query82	452	124	95	95
query83	277	244	233	233
query84	290	104	89	89
query85	776	341	306	306
query86	371	280	280	280
query87	4432	4445	4278	4278
query88	3471	2301	2284	2284
query89	401	320	331	320
query90	1831	210	203	203
query91	138	142	112	112
query92	76	65	64	64
query93	1979	927	593	593
query94	666	407	305	305
query95	376	290	283	283
query96	500	565	277	277
query97	2745	2741	2703	2703
query98	233	210	204	204
query99	1416	1389	1278	1278
Total cold run time: 282250 ms
Total hot run time: 192566 ms

doris-robot · 2025-06-09T07:37:09Z

ClickBench: Total hot run time: 29.17 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 34c78f15f5010cc145c15121febca4fe028b6aaf, data reload: false
query1	0.04	0.03	0.03
query2	0.13	0.11	0.10
query3	0.26	0.19	0.20
query4	1.59	0.19	0.11
query5	0.45	0.41	0.42
query6	1.18	0.65	0.66
query7	0.02	0.01	0.02
query8	0.04	0.04	0.04
query9	0.57	0.53	0.51
query10	0.56	0.57	0.56
query11	0.15	0.11	0.12
query12	0.15	0.12	0.12
query13	0.62	0.61	0.60
query14	0.78	0.79	0.80
query15	0.87	0.86	0.87
query16	0.39	0.38	0.38
query17	1.01	1.01	1.04
query18	0.22	0.21	0.21
query19	1.92	1.82	1.84
query20	0.02	0.01	0.01
query21	15.41	0.88	0.56
query22	0.75	1.06	0.60
query23	15.16	1.35	0.64
query24	7.35	0.86	0.80
query25	0.48	0.17	0.09
query26	0.57	0.16	0.14
query27	0.06	0.05	0.04
query28	9.18	0.89	0.44
query29	12.59	4.14	3.51
query30	0.26	0.09	0.07
query31	2.82	0.59	0.39
query32	3.23	0.55	0.48
query33	2.99	3.06	3.09
query34	15.74	5.12	4.48
query35	4.56	4.54	4.59
query36	0.68	0.50	0.48
query37	0.08	0.06	0.07
query38	0.05	0.04	0.04
query39	0.03	0.03	0.03
query40	0.16	0.14	0.13
query41	0.08	0.02	0.02
query42	0.04	0.03	0.02
query43	0.04	0.04	0.03
Total cold run time: 103.28 s
Total hot run time: 29.17 s

morningman · 2025-06-10T05:41:34Z

run buildall

doris-robot · 2025-06-10T06:16:25Z

TPC-H: Total hot run time: 33892 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit c99653d2709b22d23b93f2fe3ce7c37b816777af, data reload: false
------ Round 1 ----------------------------------
q1	26208	4967	4967	4967
q2	1961	271	189	189
q3	10271	1248	687	687
q4	10220	984	505	505
q5	7554	2322	2333	2322
q6	179	166	131	131
q7	904	735	626	626
q8	9320	1231	1120	1120
q9	7069	5190	5188	5188
q10	6907	2306	1905	1905
q11	477	286	281	281
q12	345	345	214	214
q13	17775	3748	3062	3062
q14	247	227	210	210
q15	574	492	480	480
q16	430	446	385	385
q17	593	855	362	362
q18	7537	7192	7189	7189
q19	1819	949	542	542
q20	324	332	219	219
q21	3745	3142	2334	2334
q22	1034	1033	974	974
Total cold run time: 115493 ms
Total hot run time: 33892 ms
----- Round 2, with runtime_filter_mode=off -----
q1	5306	5081	5103	5081
q2	234	315	220	220
q3	2120	2642	2311	2311
q4	1394	1815	1377	1377
q5	4518	4385	4409	4385
q6	213	169	122	122
q7	2055	1912	1771	1771
q8	2552	2605	2447	2447
q9	7214	7135	6913	6913
q10	2983	3255	2736	2736
q11	567	510	493	493
q12	681	792	619	619
q13	3501	3977	3314	3314
q14	289	287	282	282
q15	513	465	476	465
q16	433	510	469	469
q17	1161	1522	1400	1400
q18	7645	7521	7506	7506
q19	792	875	1048	875
q20	1968	2063	1926	1926
q21	4853	4419	4473	4419
q22	1098	1111	1036	1036
Total cold run time: 52090 ms
Total hot run time: 50167 ms

doris-robot · 2025-06-10T06:28:21Z

TPC-DS: Total hot run time: 193228 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit c99653d2709b22d23b93f2fe3ce7c37b816777af, data reload: false
query1	1411	1122	1089	1089
query2	6201	1872	1866	1866
query3	11174	4645	4652	4645
query4	25989	24085	23092	23092
query5	5398	653	463	463
query6	315	224	210	210
query7	3993	496	285	285
query8	280	234	218	218
query9	8519	2650	2654	2650
query10	519	333	286	286
query11	15218	15050	14860	14860
query12	163	117	116	116
query13	1575	565	424	424
query14	9474	6268	6223	6223
query15	201	205	177	177
query16	7624	671	542	542
query17	1172	768	644	644
query18	2090	428	341	341
query19	223	189	222	189
query20	126	135	120	120
query21	211	127	112	112
query22	4381	4444	4217	4217
query23	34930	33928	33689	33689
query24	8997	2413	2443	2413
query25	527	484	432	432
query26	882	269	157	157
query27	2975	520	346	346
query28	4591	2166	2175	2166
query29	649	586	463	463
query30	276	223	192	192
query31	933	828	767	767
query32	74	70	65	65
query33	541	374	321	321
query34	799	886	513	513
query35	800	866	776	776
query36	979	988	925	925
query37	110	107	79	79
query38	4219	4298	4221	4221
query39	1567	1457	1466	1457
query40	208	124	106	106
query41	62	60	57	57
query42	122	116	110	110
query43	493	534	490	490
query44	1360	850	828	828
query45	187	178	174	174
query46	859	1019	652	652
query47	1803	1892	1760	1760
query48	404	422	337	337
query49	694	514	415	415
query50	693	690	431	431
query51	4188	4311	4253	4253
query52	111	116	108	108
query53	242	263	190	190
query54	591	590	520	520
query55	94	84	85	84
query56	323	323	324	323
query57	1237	1254	1146	1146
query58	296	279	256	256
query59	2649	2767	2631	2631
query60	348	332	343	332
query61	137	157	122	122
query62	778	741	674	674
query63	229	198	193	193
query64	3452	1059	738	738
query65	4366	4334	4210	4210
query66	803	396	312	312
query67	15810	15804	15353	15353
query68	8743	907	523	523
query69	521	306	274	274
query70	1229	1133	1113	1113
query71	487	321	302	302
query72	5788	4761	4851	4761
query73	704	625	354	354
query74	9199	8930	8973	8930
query75	4004	3177	2717	2717
query76	3712	1184	767	767
query77	796	387	295	295
query78	10105	10230	9444	9444
query79	2575	834	608	608
query80	674	536	453	453
query81	478	252	219	219
query82	439	130	97	97
query83	281	253	245	245
query84	290	111	86	86
query85	750	371	397	371
query86	334	302	292	292
query87	4465	4445	4326	4326
query88	3101	2267	2241	2241
query89	413	325	290	290
query90	1949	215	214	214
query91	139	146	113	113
query92	81	62	59	59
query93	1151	943	585	585
query94	661	473	305	305
query95	376	296	285	285
query96	499	567	280	280
query97	2711	2795	2633	2633
query98	252	213	200	200
query99	1432	1429	1290	1290
Total cold run time: 280984 ms
Total hot run time: 193228 ms

doris-robot · 2025-06-10T06:33:42Z

ClickBench: Total hot run time: 29.21 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit c99653d2709b22d23b93f2fe3ce7c37b816777af, data reload: false
query1	0.04	0.04	0.03
query2	0.12	0.10	0.11
query3	0.25	0.19	0.20
query4	1.60	0.20	0.11
query5	0.43	0.41	0.41
query6	1.16	0.66	0.66
query7	0.03	0.02	0.01
query8	0.04	0.04	0.04
query9	0.58	0.53	0.52
query10	0.57	0.57	0.55
query11	0.16	0.12	0.11
query12	0.15	0.12	0.12
query13	0.63	0.60	0.59
query14	0.79	0.80	0.82
query15	0.87	0.84	0.87
query16	0.37	0.40	0.38
query17	1.08	1.05	1.06
query18	0.23	0.21	0.21
query19	1.90	1.76	1.87
query20	0.01	0.01	0.01
query21	15.40	0.88	0.54
query22	0.76	1.24	0.81
query23	14.74	1.38	0.63
query24	6.80	1.85	0.76
query25	0.46	0.22	0.14
query26	0.63	0.16	0.14
query27	0.06	0.05	0.05
query28	9.71	0.86	0.45
query29	12.58	3.99	3.34
query30	0.25	0.10	0.07
query31	2.81	0.61	0.38
query32	3.23	0.56	0.47
query33	3.04	3.09	3.07
query34	15.82	5.13	4.48
query35	4.57	4.58	4.55
query36	0.66	0.51	0.48
query37	0.09	0.07	0.07
query38	0.05	0.04	0.04
query39	0.03	0.03	0.03
query40	0.16	0.15	0.14
query41	0.09	0.03	0.03
query42	0.03	0.02	0.02
query43	0.03	0.04	0.03
Total cold run time: 103.01 s
Total hot run time: 29.21 s

…che#51471) Add new jdbc catalog property: `function_rules`. This property will enable user to customize which function can be pushed down to jdbc datasource, and which function should be rewrote to function in jdbc datasource: eg: ``` create catalog jdbc properties ( ... 'function_rules' = '{"pushdown" : {"supported": ["to_date"], "unsupported" : ["abs"]}, "rewrite" : {"to_date" : "date2"}}' ) ``` The value is a json format config: - `pushdown`: the value of `pushdown` is a map, the `supported` array indicate the function that can be pushed down to datasource. the `unsupported` array indicate the function that can not. - `rewrite`: the value of `rewrite` is map, if the function can be pushed down to datasource, it will be renamed to the value in rewrite map. eg: ``` select * from jdbc_table where abs(k1) > 0 and to_date("k2") = "2022-10-10" ``` will generate sql: ``` select * from jdbc_table where date2("k2") = "2022-10-10" ``` You can see the `abs()` is not pushed down, and `to_date()` is rewrote to `date2()`. ------- This change is compatible with previous version. Previously, there are some hard coded function rules, and also a FE config `jdbc_mysql_unsupported_pushdown_functions`. Doris will merge the user specified `function_rules` and the above rules.

) (#51918) bp #51471

…#51541) This may cause some statistic collection task fail: ``` 2025-06-05 15:02:32,277 WARN (Statistics Job Appender|182) [ExternalCatalog.buildDbForInit():902] Failed to check db DORIS_TEST exist in remote system, ignore it. java.lang.NullPointerException: Cannot invoke "org.apache.doris.datasource.jdbc.client.JdbcClient.getDatabaseNameList()" because "this.jdbcClient" is null at org.apache.doris.datasource.jdbc.JdbcExternalCatalog.listDatabaseNames(JdbcExternalCatalog.java:267) ~[doris-fe.jar:1.2-SNAPSHOT] at org.apache.doris.datasource.ExternalCatalog.getFilteredDatabaseNames(ExternalCatalog.java:476) ~[doris-fe.jar:1.2-SNAPSHOT] at org.apache.doris.datasource.ExternalCatalog.lambda$makeSureInitialized$0(ExternalCatalog.java:318) ~[doris-fe.jar:1.2-SNAPSHOT] at com.github.benmanes.caffeine.cache.LocalLoadingCache.lambda$newMappingFunction$2(LocalLoadingCache.java:145) ~[hive-catalog-shade-2.1.4.jar:2.1.4] at com.github.benmanes.caffeine.cache.LocalCache.lambda$statsAware$0(LocalCache.java:139) ~[hive-catalog-shade-2.1.4.jar:2.1.4] at com.github.benmanes.caffeine.cache.BoundedLocalCache.lambda$doComputeIfAbsent$14(BoundedLocalCache.java:2406) ~[hive-catalog-shade-2.1.4.jar:2.1.4] at java.util.concurrent.ConcurrentHashMap.compute(ConcurrentHashMap.java:1916) ~[?:?] ``` Only for branch-2.1. Fix master in #51471

…che#51471) Add new jdbc catalog property: `function_rules`. This property will enable user to customize which function can be pushed down to jdbc datasource, and which function should be rewrote to function in jdbc datasource: eg: ``` create catalog jdbc properties ( ... 'function_rules' = '{"pushdown" : {"supported": ["to_date"], "unsupported" : ["abs"]}, "rewrite" : {"to_date" : "date2"}}' ) ``` The value is a json format config: - `pushdown`: the value of `pushdown` is a map, the `supported` array indicate the function that can be pushed down to datasource. the `unsupported` array indicate the function that can not. - `rewrite`: the value of `rewrite` is map, if the function can be pushed down to datasource, it will be renamed to the value in rewrite map. eg: ``` select * from jdbc_table where abs(k1) > 0 and to_date("k2") = "2022-10-10" ``` will generate sql: ``` select * from jdbc_table where date2("k2") = "2022-10-10" ``` You can see the `abs()` is not pushed down, and `to_date()` is rewrote to `date2()`. ------- This change is compatible with previous version. Previously, there are some hard coded function rules, and also a FE config `jdbc_mysql_unsupported_pushdown_functions`. Doris will merge the user specified `function_rules` and the above rules.

…e Lombok annotations (apache#51787) Related PR: apache#51471

…g properties #51471 #51787 (#52467) bp #51471 #51787 --------- Co-authored-by: zy-kkk <zhongyk10@gmail.com>

#52817) Followup #52561 Also fix a bug introduced by #51471, which lost the function rules in jdbc external table.

apache#52817) Followup apache#52561 Also fix a bug introduced by apache#51471, which lost the function rules in jdbc external table.

…e Lombok annotations (apache#51787) Related PR: apache#51471

morningman added usercase Important user case type label dev/3.0.x labels Jun 4, 2025

morningman marked this pull request as ready for review June 4, 2025 10:22

This was referenced Jun 6, 2025
branch-3.0: [fix](jdbc) make sure init the jdbc client before using it #51540
Merged
branch-2.1: [fix](jdbc)make sure init the jdbc client before using it #51541
Merged

morningman force-pushed the custome_pushdown branch from ddc3281 to 34c78f1Compare June 9, 2025 06:42

morningman force-pushed the custome_pushdown branch from 34c78f1 to c99653dCompare June 10, 2025 05:41

morningman added 4 commits June 12, 2025 10:48

[feat](jdbc) support custom function rules in catalog properties
9af6f4a

2
3f53cc9

2
9297896

3
37eb1d2

dataroaring removed dev/3.0.x dev/3.0.x-conflict labels Jun 19, 2025

morrySnow added the dev/3.1.x label Jun 19, 2025

github-actionsBot added the dev/3.1.x-conflict label Jun 19, 2025

morningman added dev/3.0.x and removed dev/3.0.7-merged dev/3.1.x-conflict labels Jun 19, 2025

github-actionsBot added the dev/3.0.x-conflict label Jun 19, 2025

dataroaring pushed a commit that referenced this pull request Jun 20, 2025
[feat](jdbc) support custom function rules in catalog properties (#51471
459632e
) (#51918) bp #51471

dataroaring added dev/3.0.7-merged and removed dev/3.0.x dev/3.0.x-conflict labels Jun 20, 2025

morrySnow added the dev/3.1.x-conflict label Jun 20, 2025

morningman pushed a commit to morningman/doris that referenced this pull request Jun 29, 2025
[fix](catalog) synchronize reset methods in catalog classes and remov…
584a370
…e Lombok annotations (apache#51787) Related PR: apache#51471

morningman mentioned this pull request Jun 29, 2025
branch-3.1: [feat][fix](jdbc) support custom function rules in catalog properties #51471 #51787 #52467
Merged

morrySnow pushed a commit that referenced this pull request Jun 30, 2025
branch-3.1: [feat][fix](jdbc) support custom function rules in catalo…
7d372b1
…g properties #51471 #51787 (#52467) bp #51471 #51787 --------- Co-authored-by: zy-kkk <zhongyk10@gmail.com>

morrySnow added dev/3.1.0-merged and removed dev/3.1.x dev/3.1.x-conflict labels Jun 30, 2025

morningman mentioned this pull request Jul 6, 2025
[fix](test)(jdbc) fix some external regression test cases and jdbc bug #52817
Merged

morningman added a commit that referenced this pull request Jul 6, 2025
[fix](test)(jdbc) fix some external regression test cases and jdbc bug (
ebc63ea
#52817) Followup #52561 Also fix a bug introduced by #51471, which lost the function rules in jdbc external table.

zy-kkk added a commit to zy-kkk/doris that referenced this pull request Jul 13, 2025
[fix](catalog) synchronize reset methods in catalog classes and remov…
249e914
…e Lombok annotations (apache#51787) Related PR: apache#51471

zy-kkk added a commit to zy-kkk/doris that referenced this pull request Jul 13, 2025
[fix](catalog) synchronize reset methods in catalog classes and remov…
4760f70
…e Lombok annotations (apache#51787) Related PR: apache#51471

Uh oh!

Conversation

morningman commented Jun 4, 2025• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What problem does this PR solve?

Release note

Check List (For Author)

Check List (For Reviewer who merge this PR)

Uh oh!

hello-stephen commented Jun 4, 2025

Uh oh!

morningman commented Jun 4, 2025

Uh oh!

morningman commented Jun 4, 2025

Uh oh!

doris-robot commented Jun 4, 2025

Uh oh!

doris-robot commented Jun 4, 2025

Uh oh!

doris-robot commented Jun 4, 2025

Uh oh!

morningman commented Jun 6, 2025

Uh oh!

morningman commented Jun 6, 2025

Uh oh!

doris-robot commented Jun 6, 2025

Uh oh!

doris-robot commented Jun 6, 2025

Uh oh!

doris-robot commented Jun 6, 2025

Uh oh!

morningman commented Jun 9, 2025

Uh oh!

doris-robot commented Jun 9, 2025

Uh oh!

doris-robot commented Jun 9, 2025

Uh oh!

doris-robot commented Jun 9, 2025

Uh oh!

morningman commented Jun 10, 2025

Uh oh!

doris-robot commented Jun 10, 2025

Uh oh!

doris-robot commented Jun 10, 2025

Uh oh!

doris-robot commented Jun 10, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

morningman commented Jun 4, 2025•
edited
Loading