[Fix](hdfs-fs)The cache expiration should explicitly release the held fs by CalvinKirs · Pull Request #38610 · apache/doris

CalvinKirs · 2024-07-31T10:10:09Z

Proposed changes

The RemoteFSPhantomManager class is responsible for managing phantom references of RemoteFileSystem objects, ensuring that the associated FileSystem resources are automatically cleaned up when RemoteFileSystem objects are garbage collected.

Key features:

Phantom Reference Monitoring: The class uses a ReferenceQueue and PhantomReference to track RemoteFileSystem objects. When these objects are no longer in use and garbage collected, the class ensures the corresponding FileSystem resources are properly closed to prevent resource leaks.
Thread-safe Cleanup: It provides a thread-safe mechanism to start a cleanup thread only once. This thread runs periodically, checking the ReferenceQueue and closing any unused FileSystem resources.
Resource Management: The class maintains a map between phantom references and their corresponding FileSystem objects, ensuring that these resources are cleaned up appropriately.
The cleanup thread runs at regular intervals, ensuring that any RemoteFileSystem object that is no longer in use is safely removed along with its associated FileSystem resources.

doris-robot · 2024-07-31T10:10:14Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

morningman

LGTM

github-actions · 2024-07-31T10:36:38Z

PR approved by at least one committer and no changes requested.

github-actions · 2024-07-31T10:36:40Z

PR approved by anyone and no changes requested.

CalvinKirs · 2024-07-31T10:37:46Z

run buildall

doris-robot · 2024-07-31T11:50:28Z

TPC-H: Total hot run time: 42018 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 2a4fafb5c1b7238d1f29d766f7dc6101c2e349e2, data reload: false
------ Round 1 ----------------------------------
q1	17607	4181	4088	4088
q2	2041	213	215	213
q3	10600	1340	1379	1340
q4	10269	786	919	786
q5	7644	3075	3016	3016
q6	223	142	138	138
q7	1067	626	627	626
q8	9452	1908	1989	1908
q9	8533	6641	6642	6641
q10	8731	3858	3860	3858
q11	424	254	262	254
q12	415	230	225	225
q13	17762	2982	2959	2959
q14	270	241	246	241
q15	534	490	494	490
q16	497	397	390	390
q17	989	949	922	922
q18	8095	7338	7240	7240
q19	1381	1248	1224	1224
q20	571	332	327	327
q21	5507	4867	4843	4843
q22	369	293	289	289
Total cold run time: 112981 ms
Total hot run time: 42018 ms
----- Round 2, with runtime_filter_mode=off -----
q1	4081	4022	4022	4022
q2	341	230	218	218
q3	3020	3066	3162	3066
q4	2017	2039	1987	1987
q5	5870	5509	5510	5509
q6	218	131	130	130
q7	2161	1836	1849	1836
q8	3378	3442	3421	3421
q9	8709	8772	8841	8772
q10	3942	4126	3977	3977
q11	564	482	471	471
q12	774	589	590	589
q13	16273	3173	3099	3099
q14	300	288	282	282
q15	544	495	487	487
q16	467	425	422	422
q17	1788	1735	1754	1735
q18	8253	7877	7686	7686
q19	1735	1751	1730	1730
q20	2116	1870	1834	1834
q21	5860	5522	5419	5419
q22	529	459	490	459
Total cold run time: 72940 ms
Total hot run time: 57151 ms

doris-robot · 2024-07-31T12:01:17Z

TPC-DS: Total hot run time: 170660 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 2a4fafb5c1b7238d1f29d766f7dc6101c2e349e2, data reload: false
query1	910	384	365	365
query2	6450	1694	1654	1654
query3	6671	212	221	212
query4	19717	17663	17218	17218
query5	3657	515	520	515
query6	268	171	164	164
query7	4598	307	299	299
query8	248	192	188	188
query9	8529	2437	2450	2437
query10	428	289	268	268
query11	10623	10099	9818	9818
query12	127	90	94	90
query13	1642	386	376	376
query14	9738	7703	7696	7696
query15	206	165	171	165
query16	6857	476	459	459
query17	969	567	566	566
query18	1926	302	286	286
query19	193	149	148	148
query20	90	88	90	88
query21	205	100	100	100
query22	4240	3918	3927	3918
query23	33869	33767	33664	33664
query24	10303	3108	3109	3108
query25	679	434	422	422
query26	1727	156	160	156
query27	2953	282	290	282
query28	7599	2089	2062	2062
query29	1330	430	443	430
query30	250	164	154	154
query31	941	764	777	764
query32	110	58	57	57
query33	679	319	313	313
query34	939	489	505	489
query35	874	761	762	761
query36	1046	892	905	892
query37	296	81	87	81
query38	2896	2872	2805	2805
query39	883	794	801	794
query40	252	116	112	112
query41	44	43	43	43
query42	124	105	95	95
query43	455	414	417	414
query44	1183	728	742	728
query45	217	178	175	175
query46	1080	823	786	786
query47	1790	1725	1692	1692
query48	362	304	286	286
query49	943	414	410	410
query50	928	440	442	440
query51	6809	6706	6720	6706
query52	106	90	91	90
query53	252	182	173	173
query54	604	466	454	454
query55	84	77	79	77
query56	290	259	265	259
query57	1170	1042	1063	1042
query58	269	291	292	291
query59	2691	2415	2384	2384
query60	296	285	273	273
query61	109	96	95	95
query62	882	678	672	672
query63	210	186	184	184
query64	5642	1894	1906	1894
query65	3153	3117	3111	3111
query66	1329	333	333	333
query67	15426	14860	14750	14750
query68	5347	579	575	575
query69	649	377	327	327
query70	1169	1083	1072	1072
query71	476	284	283	283
query72	7545	2673	2499	2499
query73	922	332	328	328
query74	6005	5600	5642	5600
query75	3790	2728	2727	2727
query76	3634	1394	1389	1389
query77	555	305	312	305
query78	9533	8895	8970	8895
query79	2693	529	535	529
query80	1024	531	512	512
query81	562	227	225	225
query82	1402	139	129	129
query83	264	167	173	167
query84	279	86	81	81
query85	1549	325	307	307
query86	483	303	296	296
query87	3272	3107	3079	3079
query88	3648	2502	2468	2468
query89	402	289	299	289
query90	1805	195	194	194
query91	126	100	102	100
query92	60	51	52	51
query93	2152	621	607	607
query94	800	293	305	293
query95	381	271	275	271
query96	602	284	281	281
query97	3216	3085	3061	3061
query98	239	205	199	199
query99	1611	1288	1320	1288
Total cold run time: 268561 ms
Total hot run time: 170660 ms

doris-robot · 2024-07-31T12:06:46Z

ClickBench: Total hot run time: 30.15 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 2a4fafb5c1b7238d1f29d766f7dc6101c2e349e2, data reload: false
query1	0.04	0.03	0.03
query2	0.08	0.04	0.04
query3	0.23	0.05	0.04
query4	1.70	0.06	0.07
query5	0.48	0.49	0.49
query6	1.13	0.72	0.73
query7	0.01	0.01	0.01
query8	0.05	0.04	0.04
query9	0.57	0.52	0.52
query10	0.57	0.56	0.56
query11	0.16	0.11	0.12
query12	0.15	0.12	0.12
query13	0.61	0.59	0.60
query14	0.77	0.80	0.79
query15	0.90	0.87	0.89
query16	0.35	0.35	0.35
query17	0.99	0.98	0.98
query18	0.21	0.21	0.20
query19	1.88	1.72	1.73
query20	0.01	0.01	0.01
query21	15.42	0.79	0.66
query22	3.85	7.48	1.44
query23	18.07	1.33	1.31
query24	2.24	0.22	0.21
query25	0.19	0.08	0.07
query26	0.32	0.21	0.21
query27	0.47	0.24	0.23
query28	13.18	1.00	0.98
query29	12.55	3.31	3.30
query30	0.26	0.05	0.05
query31	2.88	0.40	0.39
query32	3.24	0.48	0.48
query33	2.93	3.01	2.93
query34	15.42	4.29	4.25
query35	4.30	4.29	4.30
query36	0.67	0.48	0.50
query37	0.19	0.16	0.16
query38	0.17	0.15	0.16
query39	0.05	0.04	0.04
query40	0.15	0.13	0.13
query41	0.10	0.06	0.05
query42	0.06	0.05	0.05
query43	0.05	0.04	0.04
Total cold run time: 107.65 s
Total hot run time: 30.15 s

CalvinKirs · 2024-08-01T02:06:27Z

run cloud_p1

CalvinKirs · 2024-08-01T02:06:38Z

run External

CalvinKirs · 2024-08-01T02:53:46Z

run buildall

doris-robot · 2024-08-01T03:32:34Z

TPC-H: Total hot run time: 41740 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit d9e3536ad4eebdd65bf77a485a0e78f21fdf8d6e, data reload: false
------ Round 1 ----------------------------------
q1	17627	4197	4083	4083
q2	2027	222	201	201
q3	10651	1374	1325	1325
q4	10390	811	907	811
q5	7655	2956	2991	2956
q6	224	139	135	135
q7	1039	604	607	604
q8	9444	1835	1945	1835
q9	8452	6609	6656	6609
q10	8709	3848	3860	3848
q11	428	246	250	246
q12	420	228	221	221
q13	17782	2925	2930	2925
q14	282	247	248	247
q15	530	497	501	497
q16	492	399	388	388
q17	964	959	912	912
q18	8543	7419	7261	7261
q19	1402	1223	1222	1222
q20	567	327	348	327
q21	5369	4793	4794	4793
q22	356	294	297	294
Total cold run time: 113353 ms
Total hot run time: 41740 ms
----- Round 2, with runtime_filter_mode=off -----
q1	4075	4050	4081	4050
q2	326	230	221	221
q3	3005	3041	3168	3041
q4	2005	1979	1934	1934
q5	5696	5445	5490	5445
q6	220	131	127	127
q7	2132	1829	1837	1829
q8	3350	3411	3402	3402
q9	8645	8664	8775	8664
q10	3910	4073	3894	3894
q11	548	464	475	464
q12	775	633	582	582
q13	12594	3074	3074	3074
q14	307	264	276	264
q15	536	504	497	497
q16	451	429	412	412
q17	1745	1727	1732	1727
q18	8214	7911	7652	7652
q19	1732	1764	1708	1708
q20	2067	1903	1842	1842
q21	5755	5447	5409	5409
q22	510	435	478	435
Total cold run time: 68598 ms
Total hot run time: 56673 ms

doris-robot · 2024-08-01T03:43:26Z

TPC-DS: Total hot run time: 169975 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit d9e3536ad4eebdd65bf77a485a0e78f21fdf8d6e, data reload: false
query1	912	379	380	379
query2	6452	1732	1699	1699
query3	6690	211	230	211
query4	20240	17164	17243	17164
query5	3667	517	512	512
query6	262	177	163	163
query7	4601	299	295	295
query8	250	213	188	188
query9	8500	2436	2400	2400
query10	431	283	262	262
query11	10545	9997	10056	9997
query12	123	86	85	85
query13	1626	385	378	378
query14	8868	6996	7557	6996
query15	200	162	159	159
query16	6844	428	462	428
query17	947	563	547	547
query18	1907	282	279	279
query19	190	149	143	143
query20	93	86	84	84
query21	201	101	106	101
query22	4179	4000	3923	3923
query23	33649	33532	33692	33532
query24	9848	3141	3149	3141
query25	676	397	409	397
query26	1490	152	154	152
query27	3300	288	288	288
query28	7598	2067	2043	2043
query29	1042	442	451	442
query30	239	156	170	156
query31	929	755	764	755
query32	107	57	65	57
query33	682	333	347	333
query34	935	506	503	503
query35	900	769	765	765
query36	1022	920	916	916
query37	182	82	79	79
query38	3029	2836	2809	2809
query39	871	819	847	819
query40	263	117	117	117
query41	47	45	49	45
query42	130	99	104	99
query43	463	447	425	425
query44	1203	732	746	732
query45	211	185	189	185
query46	1108	818	792	792
query47	1802	1699	1698	1698
query48	366	301	311	301
query49	877	443	439	439
query50	922	443	443	443
query51	6820	6715	6577	6577
query52	109	90	92	90
query53	252	188	194	188
query54	669	462	469	462
query55	81	79	79	79
query56	291	273	264	264
query57	1131	1036	1034	1034
query58	292	274	299	274
query59	2796	2525	2345	2345
query60	311	289	285	285
query61	116	125	113	113
query62	886	650	668	650
query63	220	189	186	186
query64	5900	1982	1957	1957
query65	3168	3086	3103	3086
query66	1346	348	342	342
query67	15384	14731	14917	14731
query68	4537	585	585	585
query69	696	387	324	324
query70	1176	1062	1052	1052
query71	450	288	293	288
query72	8240	2945	2474	2474
query73	795	333	332	332
query74	5921	5570	5670	5570
query75	3924	2749	2801	2749
query76	3250	1315	1418	1315
query77	697	312	322	312
query78	9620	9002	8934	8934
query79	3548	545	534	534
query80	2024	518	514	514
query81	569	220	234	220
query82	663	129	133	129
query83	269	179	180	179
query84	276	81	81	81
query85	1494	296	305	296
query86	461	291	316	291
query87	3249	3163	3152	3152
query88	4122	2491	2497	2491
query89	458	296	292	292
query90	1862	202	193	193
query91	131	100	104	100
query92	67	51	48	48
query93	5411	628	617	617
query94	858	290	287	287
query95	387	280	266	266
query96	617	282	317	282
query97	3250	3040	3054	3040
query98	236	199	200	199
query99	1603	1310	1343	1310
Total cold run time: 272252 ms
Total hot run time: 169975 ms

doris-robot · 2024-08-01T03:48:56Z

ClickBench: Total hot run time: 29.92 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit d9e3536ad4eebdd65bf77a485a0e78f21fdf8d6e, data reload: false
query1	0.04	0.03	0.03
query2	0.07	0.04	0.04
query3	0.22	0.04	0.05
query4	1.68	0.08	0.07
query5	0.49	0.48	0.48
query6	1.13	0.72	0.71
query7	0.02	0.01	0.01
query8	0.05	0.04	0.04
query9	0.56	0.50	0.51
query10	0.56	0.55	0.57
query11	0.16	0.11	0.12
query12	0.15	0.13	0.12
query13	0.62	0.60	0.59
query14	0.76	0.80	0.80
query15	0.89	0.86	0.86
query16	0.36	0.36	0.36
query17	1.05	0.98	0.97
query18	0.22	0.21	0.20
query19	1.85	1.71	1.72
query20	0.01	0.01	0.00
query21	15.41	0.77	0.65
query22	3.64	8.45	1.32
query23	17.84	1.37	1.32
query24	2.26	0.22	0.22
query25	0.18	0.07	0.08
query26	0.32	0.22	0.22
query27	0.46	0.23	0.23
query28	13.17	0.99	0.98
query29	12.83	3.30	3.26
query30	0.25	0.06	0.05
query31	2.88	0.42	0.40
query32	3.25	0.49	0.49
query33	2.96	2.95	2.95
query34	15.51	4.27	4.27
query35	4.30	4.29	4.24
query36	0.68	0.47	0.49
query37	0.19	0.17	0.16
query38	0.16	0.15	0.15
query39	0.04	0.04	0.03
query40	0.17	0.13	0.13
query41	0.09	0.05	0.04
query42	0.05	0.05	0.04
query43	0.04	0.04	0.04
Total cold run time: 107.57 s
Total hot run time: 29.92 s

CalvinKirs · 2024-08-01T06:36:06Z

run buildall

doris-robot · 2024-08-01T07:15:53Z

TPC-H: Total hot run time: 41306 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit d9e3536ad4eebdd65bf77a485a0e78f21fdf8d6e, data reload: false
------ Round 1 ----------------------------------
q1	17601	4088	4043	4043
q2	2022	197	197	197
q3	10464	1325	1312	1312
q4	10172	873	911	873
q5	7622	2922	2949	2922
q6	220	136	137	136
q7	1029	606	592	592
q8	9425	1922	1929	1922
q9	8673	6579	6560	6560
q10	8780	3848	3784	3784
q11	431	246	241	241
q12	424	220	220	220
q13	17791	2900	2917	2900
q14	271	254	250	250
q15	519	495	491	491
q16	514	393	391	391
q17	949	918	861	861
q18	7919	7350	7203	7203
q19	1430	1213	1198	1198
q20	554	323	336	323
q21	5258	4617	4740	4617
q22	338	270	272	270
Total cold run time: 112406 ms
Total hot run time: 41306 ms
----- Round 2, with runtime_filter_mode=off -----
q1	4019	3988	3992	3988
q2	321	225	223	223
q3	2975	2973	3041	2973
q4	1975	1968	2007	1968
q5	5503	5499	5424	5424
q6	232	131	128	128
q7	2098	1735	1846	1735
q8	3309	3380	3321	3321
q9	8643	8637	8658	8637
q10	3999	3966	4061	3966
q11	547	442	449	442
q12	721	589	575	575
q13	15761	3092	3082	3082
q14	302	267	275	267
q15	530	478	502	478
q16	459	416	422	416
q17	1755	1742	1716	1716
q18	8109	7749	7593	7593
q19	1703	1732	1724	1724
q20	2055	1836	1811	1811
q21	5669	5385	5462	5385
q22	564	470	458	458
Total cold run time: 71249 ms
Total hot run time: 56310 ms

doris-robot · 2024-08-01T07:26:37Z

TPC-DS: Total hot run time: 170308 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit d9e3536ad4eebdd65bf77a485a0e78f21fdf8d6e, data reload: false
query1	904	378	366	366
query2	6467	1677	1643	1643
query3	6665	214	229	214
query4	20251	17564	17234	17234
query5	3647	498	504	498
query6	284	158	157	157
query7	4583	300	285	285
query8	248	192	189	189
query9	8520	2330	2315	2315
query10	429	290	270	270
query11	10655	10051	10069	10051
query12	121	98	88	88
query13	1645	377	371	371
query14	8859	8009	8183	8009
query15	205	162	170	162
query16	6776	440	458	440
query17	965	569	545	545
query18	1882	310	286	286
query19	196	147	151	147
query20	91	88	87	87
query21	211	104	102	102
query22	4428	4247	4055	4055
query23	33665	33610	33434	33434
query24	9311	3086	3220	3086
query25	662	385	413	385
query26	1438	150	154	150
query27	3007	285	277	277
query28	7546	2010	1999	1999
query29	1073	441	430	430
query30	237	156	154	154
query31	939	752	755	752
query32	102	56	58	56
query33	677	298	314	298
query34	925	492	553	492
query35	851	780	770	770
query36	1042	886	874	874
query37	172	81	81	81
query38	3058	2839	2868	2839
query39	919	813	828	813
query40	252	111	109	109
query41	47	43	47	43
query42	115	97	96	96
query43	484	399	420	399
query44	1165	722	716	716
query45	206	183	179	179
query46	1098	783	771	771
query47	1795	1707	1742	1707
query48	355	286	285	285
query49	871	411	405	405
query50	911	422	433	422
query51	6844	6617	6605	6605
query52	99	91	87	87
query53	250	175	175	175
query54	620	449	432	432
query55	77	74	75	74
query56	278	256	247	247
query57	1126	1050	1035	1035
query58	267	265	278	265
query59	2505	2362	2222	2222
query60	324	275	274	274
query61	97	94	95	94
query62	865	679	665	665
query63	215	181	185	181
query64	5589	1922	1867	1867
query65	3150	3081	3079	3079
query66	1003	322	336	322
query67	15256	14898	14889	14889
query68	5761	568	574	568
query69	705	395	301	301
query70	1134	1049	1064	1049
query71	465	272	266	266
query72	7682	2691	2518	2518
query73	914	326	333	326
query74	6045	5678	5756	5678
query75	3585	2745	2733	2733
query76	3292	1386	1415	1386
query77	572	307	302	302
query78	9461	8908	9050	8908
query79	2146	535	535	535
query80	2074	496	513	496
query81	567	227	218	218
query82	821	131	127	127
query83	291	171	166	166
query84	268	80	76	76
query85	1442	315	303	303
query86	470	317	297	297
query87	3235	3067	3133	3067
query88	3715	2369	2382	2369
query89	384	287	282	282
query90	1767	189	187	187
query91	121	97	102	97
query92	59	49	48	48
query93	2379	616	612	612
query94	767	285	295	285
query95	390	286	263	263
query96	607	284	278	278
query97	3244	3044	3082	3044
query98	227	201	199	199
query99	1702	1275	1276	1275
Total cold run time: 265812 ms
Total hot run time: 170308 ms

doris-robot · 2024-08-01T07:32:07Z

ClickBench: Total hot run time: 29.81 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit d9e3536ad4eebdd65bf77a485a0e78f21fdf8d6e, data reload: false
query1	0.04	0.04	0.03
query2	0.07	0.04	0.04
query3	0.23	0.05	0.05
query4	1.67	0.06	0.07
query5	0.48	0.47	0.48
query6	1.13	0.71	0.71
query7	0.02	0.01	0.01
query8	0.06	0.04	0.04
query9	0.57	0.51	0.51
query10	0.56	0.59	0.55
query11	0.16	0.11	0.11
query12	0.15	0.12	0.13
query13	0.61	0.60	0.60
query14	0.77	0.80	0.79
query15	0.89	0.85	0.85
query16	0.36	0.36	0.35
query17	1.03	1.00	1.01
query18	0.22	0.21	0.21
query19	1.81	1.76	1.75
query20	0.02	0.02	0.01
query21	15.39	0.77	0.65
query22	3.88	8.59	1.12
query23	18.02	1.32	1.28
query24	2.27	0.22	0.21
query25	0.18	0.09	0.08
query26	0.31	0.21	0.22
query27	0.46	0.23	0.22
query28	13.16	0.99	0.96
query29	12.55	3.32	3.32
query30	0.25	0.06	0.05
query31	2.87	0.41	0.39
query32	3.26	0.48	0.48
query33	2.92	2.96	2.97
query34	15.45	4.26	4.27
query35	4.30	4.30	4.29
query36	0.68	0.48	0.49
query37	0.18	0.16	0.16
query38	0.16	0.15	0.15
query39	0.04	0.03	0.03
query40	0.16	0.13	0.13
query41	0.09	0.05	0.04
query42	0.05	0.04	0.04
query43	0.05	0.04	0.04
Total cold run time: 107.53 s
Total hot run time: 29.81 s

CalvinKirs · 2024-08-01T09:51:52Z

run p0
run External

CalvinKirs · 2024-09-03T10:33:56Z

run buildall

CalvinKirs · 2024-09-03T10:38:43Z

run buildall

CalvinKirs · 2024-09-05T05:15:04Z

run buildall

doris-robot · 2024-09-05T05:36:12Z

TPC-H: Total hot run time: 38125 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit da6709ee58acaad4b39134b494689d1659e52c2b, data reload: false
------ Round 1 ----------------------------------
q1	18043	5575	4456	4456
q2	2180	186	180	180
q3	10465	1116	1018	1018
q4	10152	752	822	752
q5	7719	2832	2766	2766
q6	231	145	139	139
q7	960	614	616	614
q8	9334	2055	2057	2055
q9	7120	6526	6525	6525
q10	7004	2269	2188	2188
q11	456	248	258	248
q12	403	224	229	224
q13	18194	3075	3095	3075
q14	276	237	246	237
q15	549	489	480	480
q16	580	518	510	510
q17	983	718	668	668
q18	7388	6790	6993	6790
q19	1390	1012	1007	1007
q20	675	335	327	327
q21	3939	2954	2846	2846
q22	1096	1021	1020	1020
Total cold run time: 109137 ms
Total hot run time: 38125 ms
----- Round 2, with runtime_filter_mode=off -----
q1	4359	4301	4230	4230
q2	371	280	265	265
q3	2872	2656	2686	2656
q4	1900	1633	1602	1602
q5	5390	5396	5413	5396
q6	228	134	130	130
q7	2105	1685	1735	1685
q8	3184	3358	3371	3358
q9	8378	8443	8371	8371
q10	3441	3211	3189	3189
q11	588	512	513	512
q12	804	599	610	599
q13	13069	3085	3052	3052
q14	308	270	297	270
q15	525	489	473	473
q16	635	570	565	565
q17	1800	1507	1475	1475
q18	7704	7398	7331	7331
q19	1678	1566	1610	1566
q20	2076	1822	1856	1822
q21	5369	5180	5210	5180
q22	1136	1080	1033	1033
Total cold run time: 67920 ms
Total hot run time: 54760 ms

doris-robot · 2024-09-05T05:48:01Z

TPC-DS: Total hot run time: 186322 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit da6709ee58acaad4b39134b494689d1659e52c2b, data reload: false
query1	913	384	380	380
query2	6479	1947	1831	1831
query3	6642	213	212	212
query4	34223	23257	23017	23017
query5	4185	496	473	473
query6	262	162	156	156
query7	4583	289	291	289
query8	267	240	213	213
query9	8571	2481	2482	2481
query10	432	277	283	277
query11	17924	15012	15001	15001
query12	148	96	94	94
query13	1613	384	367	367
query14	10880	6255	7148	6255
query15	307	166	168	166
query16	7863	476	468	468
query17	1577	550	549	549
query18	1425	281	285	281
query19	273	143	146	143
query20	117	109	108	108
query21	207	106	105	105
query22	4404	4279	4146	4146
query23	34064	33303	33678	33303
query24	11078	2866	2819	2819
query25	632	396	408	396
query26	1200	154	157	154
query27	2587	280	279	279
query28	7391	2061	2056	2056
query29	808	447	425	425
query30	310	156	159	156
query31	969	731	776	731
query32	90	55	55	55
query33	765	292	285	285
query34	991	480	478	478
query35	858	729	721	721
query36	1075	940	938	938
query37	171	94	93	93
query38	3931	3884	3814	3814
query39	1454	1402	1407	1402
query40	208	116	116	116
query41	50	49	48	48
query42	115	95	98	95
query43	484	461	469	461
query44	1251	771	737	737
query45	200	167	168	167
query46	1090	756	743	743
query47	1899	1792	1826	1792
query48	387	311	312	311
query49	1115	448	455	448
query50	820	401	397	397
query51	7044	6881	6827	6827
query52	98	88	91	88
query53	258	181	185	181
query54	1045	465	472	465
query55	78	74	74	74
query56	279	277	268	268
query57	1227	1084	1054	1054
query58	241	290	251	251
query59	2910	2832	2689	2689
query60	305	268	268	268
query61	121	119	118	118
query62	822	646	640	640
query63	216	188	183	183
query64	5435	764	732	732
query65	3267	3152	3177	3152
query66	1349	352	346	346
query67	15640	15194	15451	15194
query68	4578	569	574	569
query69	427	295	279	279
query70	1211	1011	1116	1011
query71	338	271	272	271
query72	6758	3998	3959	3959
query73	756	322	318	318
query74	9232	8900	8787	8787
query75	3393	2677	2702	2677
query76	2483	1009	955	955
query77	472	300	315	300
query78	10030	9115	8940	8940
query79	2223	527	534	527
query80	1315	497	500	497
query81	592	233	237	233
query82	926	146	147	146
query83	220	153	166	153
query84	225	75	76	75
query85	1340	314	271	271
query86	446	297	313	297
query87	4371	4270	4336	4270
query88	3721	2345	2355	2345
query89	391	279	273	273
query90	1819	188	185	185
query91	132	102	100	100
query92	61	47	47	47
query93	1691	550	550	550
query94	949	276	287	276
query95	342	248	253	248
query96	596	265	271	265
query97	3180	3100	3075	3075
query98	216	200	201	200
query99	1539	1261	1265	1261
Total cold run time: 295596 ms
Total hot run time: 186322 ms

doris-robot · 2024-09-05T05:53:36Z

ClickBench: Total hot run time: 31.42 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit da6709ee58acaad4b39134b494689d1659e52c2b, data reload: false
query1	0.05	0.04	0.04
query2	0.08	0.03	0.03
query3	0.23	0.06	0.05
query4	1.68	0.09	0.09
query5	0.50	0.48	0.48
query6	1.14	0.73	0.73
query7	0.02	0.01	0.01
query8	0.06	0.04	0.05
query9	0.54	0.48	0.50
query10	0.55	0.56	0.55
query11	0.14	0.11	0.12
query12	0.16	0.12	0.13
query13	0.60	0.59	0.59
query14	2.05	2.12	2.12
query15	0.84	0.82	0.81
query16	0.38	0.38	0.35
query17	0.95	0.97	1.03
query18	0.21	0.20	0.20
query19	1.81	1.74	1.72
query20	0.01	0.00	0.01
query21	15.40	0.66	0.65
query22	4.62	7.11	1.43
query23	18.27	1.37	1.31
query24	2.12	0.24	0.21
query25	0.16	0.08	0.08
query26	0.28	0.17	0.19
query27	0.08	0.08	0.07
query28	13.21	1.01	1.00
query29	12.66	3.28	3.26
query30	0.24	0.06	0.06
query31	2.88	0.40	0.38
query32	3.27	0.49	0.48
query33	2.96	3.00	3.06
query34	17.08	4.39	4.38
query35	4.38	4.41	4.46
query36	0.66	0.50	0.47
query37	0.19	0.15	0.16
query38	0.15	0.15	0.15
query39	0.05	0.04	0.04
query40	0.15	0.12	0.12
query41	0.09	0.05	0.04
query42	0.06	0.05	0.04
query43	0.04	0.04	0.03
Total cold run time: 111 s
Total hot run time: 31.42 s

CalvinKirs · 2024-09-05T06:07:53Z

run cloud_p0

CalvinKirs · 2024-09-05T06:48:08Z

run cloud_p0

CalvinKirs · 2024-09-05T08:07:21Z

run cloud_p0

morningman

LGTM

CalvinKirs · 2024-09-06T03:01:15Z

run cloud_p0

… fs (apache#38610) ## Proposed changes The RemoteFSPhantomManager class is responsible for managing phantom references of RemoteFileSystem objects, ensuring that the associated FileSystem resources are automatically cleaned up when RemoteFileSystem objects are garbage collected. Key features: - Phantom Reference Monitoring: The class uses a ReferenceQueue and PhantomReference to track RemoteFileSystem objects. When these objects are no longer in use and garbage collected, the class ensures the corresponding FileSystem resources are properly closed to prevent resource leaks. - Thread-safe Cleanup: It provides a thread-safe mechanism to start a cleanup thread only once. This thread runs periodically, checking the ReferenceQueue and closing any unused FileSystem resources. Resource Management: The class maintains a map between phantom references and their corresponding FileSystem objects, ensuring that these resources are cleaned up appropriately. - The cleanup thread runs at regular intervals, ensuring that any RemoteFileSystem object that is no longer in use is safely removed along with its associated FileSystem resources. (cherry picked from commit 922ec3a)

… fs #38610 (#40504)

… fs (#38610) ## Proposed changes The RemoteFSPhantomManager class is responsible for managing phantom references of RemoteFileSystem objects, ensuring that the associated FileSystem resources are automatically cleaned up when RemoteFileSystem objects are garbage collected. Key features: - Phantom Reference Monitoring: The class uses a ReferenceQueue and PhantomReference to track RemoteFileSystem objects. When these objects are no longer in use and garbage collected, the class ensures the corresponding FileSystem resources are properly closed to prevent resource leaks. - Thread-safe Cleanup: It provides a thread-safe mechanism to start a cleanup thread only once. This thread runs periodically, checking the ReferenceQueue and closing any unused FileSystem resources. Resource Management: The class maintains a map between phantom references and their corresponding FileSystem objects, ensuring that these resources are cleaned up appropriately. - The cleanup thread runs at regular intervals, ensuring that any RemoteFileSystem object that is no longer in use is safely removed along with its associated FileSystem resources.

… fs (apache#38610) ## Proposed changes The RemoteFSPhantomManager class is responsible for managing phantom references of RemoteFileSystem objects, ensuring that the associated FileSystem resources are automatically cleaned up when RemoteFileSystem objects are garbage collected. Key features: - Phantom Reference Monitoring: The class uses a ReferenceQueue and PhantomReference to track RemoteFileSystem objects. When these objects are no longer in use and garbage collected, the class ensures the corresponding FileSystem resources are properly closed to prevent resource leaks. - Thread-safe Cleanup: It provides a thread-safe mechanism to start a cleanup thread only once. This thread runs periodically, checking the ReferenceQueue and closing any unused FileSystem resources. Resource Management: The class maintains a map between phantom references and their corresponding FileSystem objects, ensuring that these resources are cleaned up appropriately. - The cleanup thread runs at regular intervals, ensuring that any RemoteFileSystem object that is no longer in use is safely removed along with its associated FileSystem resources. (cherry picked from commit 922ec3a)

…ase the held fs (#38610) (#41108) bp #38610

… fs (apache#38610) ## Proposed changes The RemoteFSPhantomManager class is responsible for managing phantom references of RemoteFileSystem objects, ensuring that the associated FileSystem resources are automatically cleaned up when RemoteFileSystem objects are garbage collected. Key features: - Phantom Reference Monitoring: The class uses a ReferenceQueue and PhantomReference to track RemoteFileSystem objects. When these objects are no longer in use and garbage collected, the class ensures the corresponding FileSystem resources are properly closed to prevent resource leaks. - Thread-safe Cleanup: It provides a thread-safe mechanism to start a cleanup thread only once. This thread runs periodically, checking the ReferenceQueue and closing any unused FileSystem resources. Resource Management: The class maintains a map between phantom references and their corresponding FileSystem objects, ensuring that these resources are cleaned up appropriately. - The cleanup thread runs at regular intervals, ensuring that any RemoteFileSystem object that is no longer in use is safely removed along with its associated FileSystem resources.

CalvinKirs added dev/2.1.x dev/3.0.x labels Jul 31, 2024

morningman added the dev/2.0.x label Jul 31, 2024

morningman previously approved these changes Jul 31, 2024
View reviewed changes

github-actionsBot added the approved Indicates a PR has been approved by one committer. label Jul 31, 2024

github-actionsBot added the reviewed label Jul 31, 2024

CalvinKirs dismissed morningman’s stale review via 3810637August 1, 2024 02:52

github-actionsBot added the doing label Aug 1, 2024

CalvinKirs force-pushed the master-hmc-oss branch from 3810637 to d9e3536Compare August 1, 2024 02:53

shuke987 added the p0_b label Aug 22, 2024

final version
da6709e

CalvinKirs changed the title ~~[WIP][Fix](hdfs-fs)The cache expiration should explicitly release the held fs~~[Fix](hdfs-fs)The cache expiration should explicitly release the held fsSep 5, 2024

morningman approved these changes Sep 5, 2024
View reviewed changes

JNSimba approved these changes Sep 6, 2024
View reviewed changes

CalvinKirs merged commit 922ec3a into apache:masterSep 6, 2024

CalvinKirs deleted the master-hmc-oss branch September 6, 2024 04:32

CalvinKirs mentioned this pull request Sep 7, 2024
[Fix](hdfs-fs)The cache expiration should explicitly release the held fs #38610 #40504
Merged

xiaokang pushed a commit that referenced this pull request Sep 7, 2024
[Fix](hdfs-fs)The cache expiration should explicitly release the held…
7347cbc
… fs #38610 (#40504)

xiaokang added dev/2.0.15-merged and removed dev/2.0.x doing labels Sep 7, 2024

CalvinKirs mentioned this pull request Sep 23, 2024
[branch-2.1][Fix](hdfs-fs)The cache expiration should explicitly release the held fs (#38610) #41108
Merged

morningman pushed a commit that referenced this pull request Sep 23, 2024
[branch-2.1][Fix](hdfs-fs)The cache expiration should explicitly rele…
d529f00
…ase the held fs (#38610) (#41108) bp #38610

morningman added dev/2.1.7-merged and removed dev/2.1.x labels Sep 23, 2024

dataroaring added dev/3.0.2-merged and removed dev/3.0.x labels Sep 26, 2024

gavinchou mentioned this pull request Oct 13, 2024
Release Note 3.0.2 #41558
Closed

Uh oh!

Conversation

CalvinKirs commented Jul 31, 2024• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Proposed changes

Uh oh!

doris-robot commented Jul 31, 2024

Uh oh!

morningman left a comment

Choose a reason for hiding this comment

Uh oh!

github-actionsBot commented Jul 31, 2024

Uh oh!

github-actionsBot commented Jul 31, 2024

Uh oh!

CalvinKirs commented Jul 31, 2024

Uh oh!

doris-robot commented Jul 31, 2024

Uh oh!

doris-robot commented Jul 31, 2024

Uh oh!

doris-robot commented Jul 31, 2024

Uh oh!

CalvinKirs commented Aug 1, 2024

Uh oh!

CalvinKirs commented Aug 1, 2024

Uh oh!

CalvinKirs commented Aug 1, 2024

Uh oh!

doris-robot commented Aug 1, 2024

Uh oh!

doris-robot commented Aug 1, 2024

Uh oh!

doris-robot commented Aug 1, 2024

Uh oh!

CalvinKirs commented Aug 1, 2024

Uh oh!

doris-robot commented Aug 1, 2024

Uh oh!

doris-robot commented Aug 1, 2024

Uh oh!

doris-robot commented Aug 1, 2024

Uh oh!

CalvinKirs commented Aug 1, 2024

Uh oh!

CalvinKirs commented Sep 3, 2024

Uh oh!

CalvinKirs commented Sep 3, 2024

Uh oh!

CalvinKirs commented Sep 5, 2024

Uh oh!

doris-robot commented Sep 5, 2024

Uh oh!

doris-robot commented Sep 5, 2024

Uh oh!

doris-robot commented Sep 5, 2024

Uh oh!

CalvinKirs commented Sep 5, 2024

Uh oh!

CalvinKirs commented Sep 5, 2024

Uh oh!

CalvinKirs commented Sep 5, 2024

Uh oh!

morningman left a comment

Choose a reason for hiding this comment

Uh oh!

CalvinKirs commented Sep 6, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

CalvinKirs commented Jul 31, 2024•
edited
Loading