[refactor](paimon) clean unused PaimonJniScanner code and refactor PaimonSysJniScanner by suxiaogang223 · Pull Request #54804 · apache/doris

suxiaogang223 · 2025-08-14T12:01:46Z

What problem does this PR solve?

Release note

Refactor PaimonSysJniScanner and IcebergSysJniScanner
Clean unused codes about PaimonJniScanner

Check List (For Author)

Test
- Regression test
- Unit Test
- Manual test (add detailed scripts or steps below)
- No need to test or manual test. Explain why:
  - This is a refactor/code format and no logic has been changed.
  - Previous test can cover this change.
  - No code files have been changed.
  - Other reason
Behavior changed:
- No.
- Yes.
Does this need documentation?
- No.
- Yes.

Check List (For Reviewer who merge this PR)

Confirm the release note
Confirm test cases
Confirm document
Add branch pick label

hello-stephen · 2025-08-14T12:02:16Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

What problem was fixed (it's best to include specific error reporting information). How it was fixed.
Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
What features were added. Why was this function added?
Which code was refactored and why was this part of the code refactored?
Which functions were optimized and what is the difference before and after the optimization?

suxiaogang223 · 2025-08-15T08:21:56Z

run buildall

hello-stephen · 2025-08-15T08:42:47Z

Cloud UT Coverage Report

Increment line coverage 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	83.97% (1441/1716)
Line Coverage	67.14% (24722/36821)
Region Coverage	68.26% (12314/18039)
Branch Coverage	57.89% (6483/11198)

doris-robot · 2025-08-15T09:55:47Z

TPC-H: Total hot run time: 34540 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit fa286d6cecc7ffd4a3679a5584399855335c11e0, data reload: false
------ Round 1 ----------------------------------
q1	16726	5301	5147	5147
q2	1911	301	190	190
q3	9980	1300	748	748
q4	10091	1029	537	537
q5	7738	2443	2358	2358
q6	182	161	133	133
q7	923	737	621	621
q8	9309	1482	1117	1117
q9	7061	5175	5173	5173
q10	6927	2382	1966	1966
q11	480	293	265	265
q12	346	355	216	216
q13	17766	3648	3030	3030
q14	233	240	211	211
q15	572	472	495	472
q16	439	419	396	396
q17	635	861	359	359
q18	7631	7352	7225	7225
q19	932	950	584	584
q20	337	329	216	216
q21	4022	2589	3024	2589
q22	1080	1030	987	987
Total cold run time: 105321 ms
Total hot run time: 34540 ms
----- Round 2, with runtime_filter_mode=off -----
q1	5191	5168	5172	5168
q2	244	312	213	213
q3	2183	2746	2288	2288
q4	1416	1793	1346	1346
q5	4279	4529	4464	4464
q6	213	171	133	133
q7	2007	2022	1835	1835
q8	2633	2668	2552	2552
q9	7523	7306	7375	7306
q10	3186	3336	2880	2880
q11	593	532	506	506
q12	719	921	630	630
q13	3472	3965	3340	3340
q14	289	311	286	286
q15	533	484	474	474
q16	478	507	445	445
q17	1227	1603	1377	1377
q18	7830	7746	7604	7604
q19	837	807	841	807
q20	2070	2031	1902	1902
q21	4791	4363	4302	4302
q22	1085	1073	1007	1007
Total cold run time: 52799 ms
Total hot run time: 50865 ms

doris-robot · 2025-08-15T10:07:20Z

TPC-DS: Total hot run time: 184917 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit fa286d6cecc7ffd4a3679a5584399855335c11e0, data reload: false
query1	1017	417	423	417
query2	6511	1753	1747	1747
query3	6740	226	217	217
query4	26360	23193	22983	22983
query5	4379	605	489	489
query6	295	217	200	200
query7	4625	512	296	296
query8	286	244	228	228
query9	8574	2904	2893	2893
query10	462	347	276	276
query11	15387	14950	14775	14775
query12	172	116	112	112
query13	1661	555	430	430
query14	8884	5723	5706	5706
query15	208	178	162	162
query16	7674	655	495	495
query17	1203	722	607	607
query18	2027	419	323	323
query19	199	206	173	173
query20	127	123	118	118
query21	218	124	114	114
query22	4364	4415	4098	4098
query23	34106	33319	33253	33253
query24	8123	2317	2359	2317
query25	533	468	414	414
query26	1241	276	159	159
query27	2723	502	343	343
query28	4230	2255	2224	2224
query29	772	570	445	445
query30	283	222	184	184
query31	881	789	689	689
query32	81	75	77	75
query33	551	366	340	340
query34	795	868	508	508
query35	805	814	729	729
query36	980	1014	902	902
query37	120	105	87	87
query38	4002	3999	3945	3945
query39	1455	1419	1403	1403
query40	218	126	116	116
query41	62	56	57	56
query42	121	106	119	106
query43	504	496	464	464
query44	1335	847	860	847
query45	178	173	161	161
query46	851	1016	631	631
query47	1779	1838	1716	1716
query48	385	421	330	330
query49	711	484	396	396
query50	642	697	391	391
query51	4166	4134	4138	4134
query52	111	106	109	106
query53	231	254	194	194
query54	593	584	524	524
query55	92	87	88	87
query56	304	304	291	291
query57	1170	1208	1147	1147
query58	275	271	280	271
query59	2720	2734	2626	2626
query60	344	332	330	330
query61	127	122	122	122
query62	809	729	653	653
query63	235	191	188	188
query64	4136	1011	676	676
query65	4308	4227	4212	4212
query66	1098	475	326	326
query67	15939	15469	15108	15108
query68	8249	910	572	572
query69	479	322	285	285
query70	1247	1076	1141	1076
query71	527	328	303	303
query72	5560	4797	4836	4797
query73	790	643	360	360
query74	8933	9186	8955	8955
query75	3861	3114	2623	2623
query76	3789	1153	734	734
query77	786	399	323	323
query78	9632	9683	8795	8795
query79	2499	833	597	597
query80	612	540	469	469
query81	467	254	222	222
query82	462	141	113	113
query83	278	252	232	232
query84	292	114	81	81
query85	788	368	324	324
query86	342	304	296	296
query87	4270	4257	4221	4221
query88	3312	2188	2212	2188
query89	380	321	284	284
query90	1924	218	218	218
query91	209	137	112	112
query92	87	71	67	67
query93	1858	980	630	630
query94	663	411	306	306
query95	394	317	306	306
query96	485	579	273	273
query97	2662	2684	2577	2577
query98	253	221	204	204
query99	1418	1392	1289	1289
Total cold run time: 274006 ms
Total hot run time: 184917 ms

doris-robot · 2025-08-15T10:12:56Z

ClickBench: Total hot run time: 33.32 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit fa286d6cecc7ffd4a3679a5584399855335c11e0, data reload: false
query1	0.04	0.04	0.03
query2	0.10	0.06	0.05
query3	0.30	0.07	0.06
query4	1.62	0.08	0.08
query5	0.42	0.40	0.41
query6	1.16	0.66	0.62
query7	0.03	0.02	0.03
query8	0.07	0.05	0.05
query9	0.64	0.54	0.53
query10	0.58	0.58	0.61
query11	0.26	0.12	0.13
query12	0.25	0.14	0.13
query13	0.64	0.62	0.65
query14	0.81	0.83	0.84
query15	0.98	0.87	0.87
query16	0.40	0.38	0.38
query17	1.03	1.09	1.05
query18	0.23	0.22	0.22
query19	1.99	1.85	1.84
query20	0.01	0.02	0.01
query21	15.36	1.01	0.69
query22	0.93	1.07	0.87
query23	14.75	1.49	0.75
query24	4.94	0.58	0.31
query25	0.17	0.09	0.08
query26	0.55	0.22	0.18
query27	0.10	0.09	0.09
query28	11.06	1.17	0.57
query29	12.57	3.97	3.33
query30	3.10	3.06	3.05
query31	2.81	0.62	0.42
query32	3.24	0.61	0.50
query33	3.08	3.12	3.12
query34	16.87	5.55	4.75
query35	4.91	4.89	4.90
query36	0.65	0.52	0.52
query37	0.22	0.19	0.19
query38	0.18	0.17	0.16
query39	0.06	0.04	0.05
query40	0.20	0.18	0.17
query41	0.10	0.07	0.06
query42	0.07	0.05	0.06
query43	0.06	0.05	0.04
Total cold run time: 107.54 s
Total hot run time: 33.32 s

doris-robot · 2025-08-15T11:16:48Z

BE UT Coverage Report

Increment line coverage 0.00% (0/13) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	59.55% (16796/28205)
Line Coverage	48.49% (152640/314767)
Region Coverage	37.42% (116032/310061)
Branch Coverage	40.35% (51197/126868)

hello-stephen · 2025-08-15T12:10:26Z

BE Regression && UT Coverage Report

Increment line coverage 0.00% (0/13) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	81.93% (22641/27635)
Line Coverage	74.67% (234872/314544)
Region Coverage	61.70% (194558/315304)
Branch Coverage	65.99% (84341/127803)

suxiaogang223 · 2025-08-17T13:53:35Z

run buildall

hello-stephen · 2025-08-17T14:09:05Z

Cloud UT Coverage Report

Increment line coverage 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	83.97% (1441/1716)
Line Coverage	67.21% (24785/36878)
Region Coverage	68.33% (12334/18051)
Branch Coverage	57.92% (6491/11206)

doris-robot · 2025-08-17T15:38:46Z

TPC-H: Total hot run time: 33773 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit df280e5db12053c167006f96b399f5aa9bf09eb9, data reload: false
------ Round 1 ----------------------------------
q1	17177	5279	5021	5021
q2	1936	276	178	178
q3	10116	1302	720	720
q4	10218	1027	535	535
q5	7524	2362	2334	2334
q6	179	161	131	131
q7	903	744	579	579
q8	9300	1308	1071	1071
q9	6971	5295	5042	5042
q10	6895	2384	1999	1999
q11	481	285	265	265
q12	341	356	207	207
q13	17758	3657	3039	3039
q14	231	233	210	210
q15	537	483	494	483
q16	428	416	362	362
q17	593	877	357	357
q18	7495	7196	7137	7137
q19	1100	968	572	572
q20	350	324	217	217
q21	3784	3205	2303	2303
q22	1071	1049	1011	1011
Total cold run time: 105388 ms
Total hot run time: 33773 ms
----- Round 2, with runtime_filter_mode=off -----
q1	5190	5096	5099	5096
q2	246	323	218	218
q3	2192	2674	2298	2298
q4	1330	1779	1349	1349
q5	4249	4420	4498	4420
q6	248	176	127	127
q7	2033	1940	1799	1799
q8	2670	2670	2644	2644
q9	7267	7280	7315	7280
q10	3062	3360	2857	2857
q11	590	496	488	488
q12	698	763	636	636
q13	3517	3930	3290	3290
q14	296	307	274	274
q15	534	481	464	464
q16	475	488	450	450
q17	1197	1546	1403	1403
q18	7765	7705	7627	7627
q19	836	790	982	790
q20	2000	1988	1794	1794
q21	4838	4377	4291	4291
q22	1074	1021	975	975
Total cold run time: 52307 ms
Total hot run time: 50570 ms

doris-robot · 2025-08-17T15:50:17Z

TPC-DS: Total hot run time: 184417 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit df280e5db12053c167006f96b399f5aa9bf09eb9, data reload: false
query1	997	380	422	380
query2	6536	1704	1763	1704
query3	6740	220	228	220
query4	26341	23085	22868	22868
query5	4324	585	479	479
query6	291	208	192	192
query7	4642	511	285	285
query8	279	228	225	225
query9	8595	2831	2868	2831
query10	489	354	307	307
query11	15663	15008	14772	14772
query12	162	113	115	113
query13	1654	563	409	409
query14	8685	5689	5666	5666
query15	216	190	159	159
query16	7150	648	493	493
query17	1204	732	606	606
query18	1993	412	326	326
query19	195	191	160	160
query20	126	125	122	122
query21	220	130	114	114
query22	4057	4223	4044	4044
query23	34065	33228	33210	33210
query24	7998	2332	2298	2298
query25	546	474	399	399
query26	1226	267	160	160
query27	2761	510	341	341
query28	4298	2234	2251	2234
query29	769	561	443	443
query30	296	210	194	194
query31	854	814	718	718
query32	86	72	70	70
query33	561	363	360	360
query34	795	831	503	503
query35	807	814	742	742
query36	958	1006	913	913
query37	134	101	94	94
query38	4033	4073	4028	4028
query39	1488	1433	1394	1394
query40	215	123	121	121
query41	59	54	54	54
query42	117	109	113	109
query43	512	493	463	463
query44	1351	852	852	852
query45	171	166	163	163
query46	852	1001	636	636
query47	1780	1801	1719	1719
query48	383	412	306	306
query49	738	486	379	379
query50	640	686	389	389
query51	4079	4069	4081	4069
query52	112	109	94	94
query53	228	250	200	200
query54	589	588	521	521
query55	92	111	86	86
query56	325	319	312	312
query57	1203	1218	1127	1127
query58	279	259	262	259
query59	2675	2694	2618	2618
query60	349	334	327	327
query61	129	116	122	116
query62	828	713	649	649
query63	219	200	181	181
query64	4317	1016	782	782
query65	4279	4191	4184	4184
query66	1145	411	314	314
query67	15587	15251	15109	15109
query68	8316	913	566	566
query69	481	319	294	294
query70	1239	1157	1133	1133
query71	488	319	314	314
query72	5676	4692	4808	4692
query73	755	660	350	350
query74	8837	9138	8916	8916
query75	3893	3049	2624	2624
query76	3813	1145	731	731
query77	786	402	328	328
query78	9578	9480	8855	8855
query79	2831	832	592	592
query80	614	550	551	550
query81	465	252	220	220
query82	427	135	108	108
query83	284	253	238	238
query84	293	98	84	84
query85	793	361	331	331
query86	341	303	296	296
query87	4304	4215	4093	4093
query88	3187	2162	2248	2162
query89	405	313	279	279
query90	1931	222	219	219
query91	134	136	108	108
query92	89	72	61	61
query93	1703	996	628	628
query94	678	407	314	314
query95	396	311	310	310
query96	477	571	270	270
query97	2668	2676	2520	2520
query98	234	211	206	206
query99	1445	1413	1310	1310
Total cold run time: 273048 ms
Total hot run time: 184417 ms

doris-robot · 2025-08-17T15:55:52Z

ClickBench: Total hot run time: 33.44 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit df280e5db12053c167006f96b399f5aa9bf09eb9, data reload: false
query1	0.04	0.04	0.03
query2	0.11	0.06	0.06
query3	0.29	0.07	0.06
query4	1.60	0.08	0.08
query5	0.43	0.40	0.41
query6	1.16	0.64	0.64
query7	0.02	0.02	0.02
query8	0.07	0.05	0.05
query9	0.65	0.53	0.52
query10	0.59	0.58	0.56
query11	0.26	0.13	0.13
query12	0.25	0.14	0.14
query13	0.64	0.63	0.65
query14	0.82	0.84	0.84
query15	0.99	0.87	0.88
query16	0.37	0.39	0.40
query17	1.06	1.06	1.06
query18	0.23	0.22	0.22
query19	1.93	1.86	1.88
query20	0.01	0.02	0.02
query21	15.37	1.04	0.69
query22	0.93	1.06	0.86
query23	14.69	1.51	0.78
query24	5.49	0.58	0.29
query25	0.17	0.09	0.09
query26	0.56	0.21	0.18
query27	0.10	0.09	0.10
query28	11.02	1.15	0.58
query29	12.71	4.01	3.34
query30	3.07	3.03	3.01
query31	2.82	0.62	0.42
query32	3.26	0.60	0.52
query33	3.22	3.10	3.13
query34	16.53	5.43	4.80
query35	4.84	4.88	4.91
query36	0.66	0.53	0.51
query37	0.22	0.19	0.19
query38	0.18	0.17	0.16
query39	0.05	0.04	0.05
query40	0.22	0.18	0.18
query41	0.12	0.06	0.06
query42	0.07	0.06	0.05
query43	0.06	0.06	0.04
Total cold run time: 107.88 s
Total hot run time: 33.44 s

doris-robot · 2025-08-17T16:45:58Z

BE UT Coverage Report

Increment line coverage 0.00% (0/13) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	59.63% (16772/28127)
Line Coverage	48.52% (152580/314460)
Region Coverage	37.41% (115909/309849)
Branch Coverage	40.37% (51198/126836)

hello-stephen · 2025-08-17T17:51:13Z

BE Regression && UT Coverage Report

Increment line coverage 0.00% (0/13) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	81.99% (22595/27558)
Line Coverage	74.66% (234603/314238)
Region Coverage	61.76% (194592/315093)
Branch Coverage	65.95% (84270/127771)

github-actions · 2025-08-18T00:24:17Z

PR approved by at least one committer and no changes requested.

github-actions · 2025-08-18T00:24:19Z

PR approved by anyone and no changes requested.

…imonSysJniScanner (apache#54804) 1. Refactor `PaimonSysJniScanner` and `IcebergSysJniScanner` 2. Clean unused codes about `PaimonJniScanner`

… refactor PaimonSysJniScanner (#54679 #54804) (#55020) bp: #54679 adapt session time_zone for PaimonJniScanner #54804 refactor PaimonSysJniScanner

### What problem does this PR solve? - related pr: #54804 Metadata scans (e.g., Iceberg all_files) can produce many splits, but the prior logic grouped them into at most one scan range per backend, which capped `NumScanners` and `MaxScanConcurrency` at 1 in common cases. This change aligns metadata scans with per-split assignment used by file scans, enabling higher parallelism.

suxiaogang223 added 6 commits August 14, 2025 15:08

clean code
fa743fc

change thrift
b34785b

remove PaimonTableCache
f4e2db1

change api getMetaScanRange
0a1b1c0

fix IcebergTableValuedFunction and PaimonTableValuedFunction
e8cffd2

split the tasks to be
3b98e12

suxiaogang223 marked this pull request as draft August 14, 2025 13:16

suxiaogang223and others added 5 commits August 14, 2025 21:40

clean meta_scanner and fix jni
ab3001b

fix
956664c

refactor IcebergSysTableJniScanner
015b72a

refactor PaimonSysTableJniScanner
080db78

fix build
02b66c5

suxiaogang223 marked this pull request as ready for review August 15, 2025 08:08

fix
fa286d6

morningman added the dev/3.1.x label Aug 15, 2025

morningman reviewed Aug 15, 2025
View reviewed changes

Comment threadfe/fe-core/src/main/java/org/apache/doris/datasource/tvf/source/MetadataScanNode.java Outdated

fix
df280e5

morningman approved these changes Aug 18, 2025
View reviewed changes

github-actionsBot added approved Indicates a PR has been approved by one committer. reviewed labels Aug 18, 2025

zy-kkk approved these changes Aug 18, 2025
View reviewed changes

morningman merged commit 1551b99 into apache:masterAug 18, 2025
26 of 28 checks passed

github-actionsBot added the dev/3.1.x-conflict label Aug 18, 2025

suxiaogang223 mentioned this pull request Aug 19, 2025
branch-3.1: [refactor](paimon) clean unused PaimonJniScanner code and refactor PaimonSysJniScanner (#54679 #54804) #55020
Merged

morningman added dev/3.1.0-merged and removed dev/3.1.x dev/3.1.x-conflict labels Aug 21, 2025

suxiaogang223 deleted the clean_paimon branch September 23, 2025 03:18

suxiaogang223 mentioned this pull request Jan 27, 2026
[enhance](multi-catalog) Split metadata scan ranges per split #60257
Merged
16 tasks

Uh oh!

Conversation

suxiaogang223 commented Aug 14, 2025• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What problem does this PR solve?

Release note

Check List (For Author)

Check List (For Reviewer who merge this PR)

Uh oh!

hello-stephen commented Aug 14, 2025

Uh oh!

suxiaogang223 commented Aug 15, 2025

Uh oh!

hello-stephen commented Aug 15, 2025

Cloud UT Coverage Report

Uh oh!

doris-robot commented Aug 15, 2025

Uh oh!

doris-robot commented Aug 15, 2025

Uh oh!

doris-robot commented Aug 15, 2025

Uh oh!

doris-robot commented Aug 15, 2025

BE UT Coverage Report

Uh oh!

hello-stephen commented Aug 15, 2025

BE Regression && UT Coverage Report

Uh oh!

Uh oh!

suxiaogang223 commented Aug 17, 2025

Uh oh!

hello-stephen commented Aug 17, 2025

Cloud UT Coverage Report

Uh oh!

doris-robot commented Aug 17, 2025

Uh oh!

doris-robot commented Aug 17, 2025

Uh oh!

doris-robot commented Aug 17, 2025

Uh oh!

doris-robot commented Aug 17, 2025

BE UT Coverage Report

Uh oh!

hello-stephen commented Aug 17, 2025

BE Regression && UT Coverage Report

Uh oh!

github-actionsBot commented Aug 18, 2025

Uh oh!

github-actionsBot commented Aug 18, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

suxiaogang223 commented Aug 14, 2025•
edited
Loading