[fix](split) FileSystemCacheKey are always different in overload equals by AshinGau · Pull Request #36432 · apache/doris

AshinGau · 2024-06-18T01:38:13Z

Proposed changes

Fixed Bugs introduced from #33937

FileSystemCacheKey.equals() compares properties by ==, resulting in creating new file system in each partition
dfsFileSystem is not synchronized, resulting in creating more file systems than need.
jobConf.iterator() will produce more than 2000 pairs of key-value

doris-robot · 2024-06-18T01:38:18Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

AshinGau · 2024-06-18T01:42:53Z

run buildall

AshinGau · 2024-06-18T09:24:32Z

run buildall

doris-robot · 2024-06-19T02:22:13Z

TPC-H: Total hot run time: 39501 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 55c59600addeb84a9ef1350aa76a405290c3ac50, data reload: false
------ Round 1 ----------------------------------
q1	17671	4377	4307	4307
q2	2036	199	217	199
q3	10431	1095	1069	1069
q4	10193	767	853	767
q5	7483	2620	2555	2555
q6	220	138	138	138
q7	963	604	600	600
q8	9225	2050	2024	2024
q9	8796	6431	6458	6431
q10	9026	3715	3671	3671
q11	432	233	231	231
q12	411	228	230	228
q13	17767	2993	2957	2957
q14	257	223	214	214
q15	514	486	486	486
q16	529	380	370	370
q17	963	683	654	654
q18	7960	7484	7358	7358
q19	7142	1451	1465	1451
q20	643	331	318	318
q21	4872	3829	3141	3141
q22	375	334	332	332
Total cold run time: 117909 ms
Total hot run time: 39501 ms
----- Round 2, with runtime_filter_mode=off -----
q1	4390	4273	4322	4273
q2	367	257	279	257
q3	2959	2953	2884	2884
q4	1979	1679	1695	1679
q5	5600	5486	5467	5467
q6	217	128	132	128
q7	2225	1886	1832	1832
q8	3263	3428	3439	3428
q9	8571	8701	8793	8701
q10	4099	3863	3794	3794
q11	572	491	488	488
q12	802	621	628	621
q13	16543	3172	3168	3168
q14	306	286	287	286
q15	525	480	493	480
q16	493	419	440	419
q17	1832	1497	1494	1494
q18	8094	7782	7663	7663
q19	1779	1494	1626	1494
q20	2159	1856	1852	1852
q21	5123	4751	4918	4751
q22	592	554	567	554
Total cold run time: 72490 ms
Total hot run time: 55713 ms

doris-robot · 2024-06-19T02:33:14Z

TPC-DS: Total hot run time: 170365 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 55c59600addeb84a9ef1350aa76a405290c3ac50, data reload: false
query1	923	378	382	378
query2	6432	2366	2301	2301
query3	6634	209	211	209
query4	19703	17281	17307	17281
query5	3583	474	463	463
query6	234	159	172	159
query7	4597	297	285	285
query8	300	287	259	259
query9	8417	2423	2399	2399
query10	578	304	281	281
query11	10545	10168	9938	9938
query12	121	111	83	83
query13	1628	351	357	351
query14	9879	6840	7594	6840
query15	229	184	186	184
query16	7761	261	254	254
query17	1919	557	499	499
query18	1947	273	264	264
query19	204	156	148	148
query20	89	82	84	82
query21	205	138	127	127
query22	4357	4023	4094	4023
query23	33757	33434	33625	33434
query24	10814	2982	2850	2850
query25	575	372	362	362
query26	706	152	151	151
query27	2299	326	317	317
query28	6004	2152	2096	2096
query29	906	661	624	624
query30	245	152	149	149
query31	991	782	760	760
query32	91	54	56	54
query33	670	292	268	268
query34	878	469	454	454
query35	760	641	591	591
query36	1114	966	972	966
query37	154	70	69	69
query38	2969	2908	2838	2838
query39	908	850	809	809
query40	218	129	129	129
query41	57	58	57	57
query42	111	110	106	106
query43	584	520	569	520
query44	1100	727	747	727
query45	194	164	162	162
query46	1061	710	720	710
query47	1886	1767	1793	1767
query48	375	300	291	291
query49	882	408	411	408
query50	757	384	388	384
query51	6764	6730	6699	6699
query52	103	95	96	95
query53	359	289	299	289
query54	884	468	446	446
query55	74	74	75	74
query56	290	276	272	272
query57	1119	1042	1074	1042
query58	272	267	286	267
query59	3334	3214	3238	3214
query60	315	290	299	290
query61	111	114	109	109
query62	602	456	455	455
query63	316	289	287	287
query64	8572	2335	1711	1711
query65	3183	3059	3083	3059
query66	736	319	325	319
query67	15541	14867	14854	14854
query68	6213	534	520	520
query69	693	478	382	382
query70	1097	1069	1139	1069
query71	498	277	257	257
query72	8737	5810	2789	2789
query73	760	319	318	318
query74	5862	5469	5450	5450
query75	4508	2670	2656	2656
query76	3375	1018	941	941
query77	736	294	285	285
query78	10532	9841	9887	9841
query79	4320	526	511	511
query80	1032	459	456	456
query81	553	221	226	221
query82	1268	104	101	101
query83	320	164	167	164
query84	268	87	86	86
query85	1188	283	264	264
query86	445	339	318	318
query87	3232	3077	3090	3077
query88	4785	2386	2328	2328
query89	506	387	380	380
query90	1883	190	186	186
query91	128	96	96	96
query92	60	47	47	47
query93	5213	500	498	498
query94	1151	184	182	182
query95	394	307	310	307
query96	599	263	265	263
query97	3194	3010	3027	3010
query98	213	197	189	189
query99	1181	830	846	830
Total cold run time: 278870 ms
Total hot run time: 170365 ms

doris-robot · 2024-06-19T02:38:45Z

ClickBench: Total hot run time: 30.58 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 55c59600addeb84a9ef1350aa76a405290c3ac50, data reload: false
query1	0.04	0.03	0.03
query2	0.08	0.03	0.04
query3	0.22	0.05	0.05
query4	1.68	0.07	0.08
query5	0.48	0.48	0.48
query6	1.12	0.73	0.73
query7	0.03	0.02	0.02
query8	0.05	0.05	0.05
query9	0.54	0.48	0.48
query10	0.55	0.56	0.54
query11	0.16	0.11	0.12
query12	0.14	0.12	0.11
query13	0.59	0.58	0.60
query14	0.78	0.79	0.79
query15	0.82	0.81	0.81
query16	0.36	0.36	0.36
query17	1.02	0.96	1.02
query18	0.21	0.28	0.22
query19	1.75	1.70	1.74
query20	0.01	0.02	0.01
query21	15.42	0.64	0.64
query22	4.28	7.13	2.00
query23	18.32	1.34	1.31
query24	2.18	0.22	0.22
query25	0.15	0.10	0.09
query26	0.26	0.17	0.17
query27	0.08	0.07	0.08
query28	13.24	1.02	1.01
query29	12.62	3.29	3.26
query30	0.26	0.06	0.05
query31	2.85	0.38	0.39
query32	3.29	0.47	0.46
query33	2.98	2.94	2.90
query34	17.36	4.51	4.38
query35	4.50	4.46	4.48
query36	0.65	0.46	0.49
query37	0.18	0.15	0.16
query38	0.15	0.14	0.14
query39	0.04	0.04	0.04
query40	0.17	0.14	0.14
query41	0.10	0.04	0.04
query42	0.06	0.04	0.04
query43	0.05	0.04	0.05
Total cold run time: 109.82 s
Total hot run time: 30.58 s

morningman

LGTM

github-actions · 2024-06-19T08:08:19Z

PR approved by at least one committer and no changes requested.

github-actions · 2024-06-19T08:08:21Z

PR approved by anyone and no changes requested.

This reverts commit efe8e4c.

AshinGau · 2024-06-19T08:33:05Z

run buildall

doris-robot · 2024-06-19T08:57:21Z

TPC-H: Total hot run time: 39609 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 35587318507fa41b46bbd444c3aee0efd80a86d1, data reload: false
------ Round 1 ----------------------------------
q1	17611	4870	4249	4249
q2	2032	199	204	199
q3	10517	1120	1051	1051
q4	10680	822	778	778
q5	8173	2684	2755	2684
q6	227	142	142	142
q7	963	628	624	624
q8	9556	2109	2076	2076
q9	8992	6486	6479	6479
q10	8873	3710	3696	3696
q11	483	236	243	236
q12	490	229	225	225
q13	17757	2957	2987	2957
q14	281	215	219	215
q15	515	485	467	467
q16	518	395	390	390
q17	961	659	692	659
q18	7897	7490	7350	7350
q19	2615	1407	1507	1407
q20	688	313	324	313
q21	4931	3291	3082	3082
q22	394	330	336	330
Total cold run time: 115154 ms
Total hot run time: 39609 ms
----- Round 2, with runtime_filter_mode=off -----
q1	4380	4283	4256	4256
q2	380	279	268	268
q3	2991	2752	2732	2732
q4	1913	1585	1647	1585
q5	5269	5316	5269	5269
q6	215	135	132	132
q7	2116	1763	1714	1714
q8	3161	3302	3296	3296
q9	8312	8313	8342	8313
q10	3879	3730	3697	3697
q11	587	489	488	488
q12	767	597	598	597
q13	16480	2979	3006	2979
q14	283	268	262	262
q15	519	483	478	478
q16	474	414	424	414
q17	1777	1492	1465	1465
q18	7576	7468	7451	7451
q19	5307	1579	1594	1579
q20	1973	1770	1789	1770
q21	4799	4797	4669	4669
q22	587	564	561	561
Total cold run time: 73745 ms
Total hot run time: 53975 ms

doris-robot · 2024-06-19T09:08:14Z

TPC-DS: Total hot run time: 169596 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 35587318507fa41b46bbd444c3aee0efd80a86d1, data reload: false
query1	929	373	374	373
query2	6469	2344	2146	2146
query3	6648	204	206	204
query4	19524	17365	17410	17365
query5	4204	468	466	466
query6	284	168	162	162
query7	4596	296	286	286
query8	318	276	284	276
query9	8543	2383	2345	2345
query10	600	298	289	289
query11	10663	9989	10010	9989
query12	129	95	85	85
query13	1649	375	377	375
query14	9574	7069	7764	7069
query15	280	193	194	193
query16	8149	268	264	264
query17	1904	535	536	535
query18	2095	273	274	273
query19	188	153	157	153
query20	89	86	87	86
query21	239	126	130	126
query22	4440	3963	3897	3897
query23	33877	33318	33280	33280
query24	11172	2922	2813	2813
query25	623	357	382	357
query26	1373	155	156	155
query27	2950	310	313	310
query28	7506	2018	2025	2018
query29	903	649	632	632
query30	291	150	152	150
query31	945	734	753	734
query32	94	56	58	56
query33	778	291	279	279
query34	976	477	467	467
query35	743	603	630	603
query36	1071	940	938	938
query37	153	71	75	71
query38	2888	2716	2732	2716
query39	856	778	795	778
query40	212	130	126	126
query41	55	54	53	53
query42	119	99	97	97
query43	590	523	571	523
query44	1227	723	718	718
query45	216	167	168	167
query46	1077	708	705	705
query47	1832	1762	1761	1761
query48	366	290	304	290
query49	1085	404	398	398
query50	762	382	394	382
query51	6779	6592	6624	6592
query52	104	96	103	96
query53	373	295	291	291
query54	857	443	447	443
query55	77	75	74	74
query56	284	259	270	259
query57	1169	1034	1027	1027
query58	244	244	284	244
query59	3232	3106	3311	3106
query60	293	300	285	285
query61	94	99	92	92
query62	630	433	456	433
query63	322	290	285	285
query64	9064	2254	1767	1767
query65	3217	3093	3133	3093
query66	997	378	333	333
query67	15469	14869	14911	14869
query68	4605	542	535	535
query69	463	306	306	306
query70	1207	1080	1073	1073
query71	412	269	271	269
query72	7150	2788	2632	2632
query73	747	321	340	321
query74	5890	5475	5489	5475
query75	3446	2625	2639	2625
query76	2912	1006	894	894
query77	465	297	302	297
query78	10307	9917	9681	9681
query79	2744	515	507	507
query80	2490	480	470	470
query81	556	221	219	219
query82	1030	110	105	105
query83	310	179	177	177
query84	266	92	88	88
query85	1434	295	346	295
query86	450	344	327	327
query87	3284	3087	3083	3083
query88	3645	2356	2330	2330
query89	485	382	398	382
query90	1813	202	191	191
query91	130	99	101	99
query92	68	48	51	48
query93	1825	519	508	508
query94	1225	186	184	184
query95	404	307	320	307
query96	589	271	266	266
query97	3253	3079	3039	3039
query98	221	202	195	195
query99	1310	845	863	845
Total cold run time: 274595 ms
Total hot run time: 169596 ms

doris-robot · 2024-06-19T09:13:43Z

ClickBench: Total hot run time: 30.25 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 35587318507fa41b46bbd444c3aee0efd80a86d1, data reload: false
query1	0.04	0.04	0.04
query2	0.09	0.04	0.05
query3	0.23	0.05	0.05
query4	1.68	0.07	0.08
query5	0.52	0.52	0.47
query6	1.12	0.72	0.74
query7	0.02	0.01	0.02
query8	0.06	0.04	0.04
query9	0.54	0.49	0.50
query10	0.54	0.54	0.55
query11	0.14	0.11	0.10
query12	0.15	0.12	0.12
query13	0.59	0.59	0.61
query14	0.78	0.77	0.79
query15	0.84	0.81	0.83
query16	0.36	0.35	0.36
query17	1.02	0.99	0.97
query18	0.20	0.27	0.25
query19	1.86	1.75	1.68
query20	0.02	0.01	0.02
query21	15.44	0.66	0.65
query22	4.63	6.90	1.71
query23	18.23	1.40	1.22
query24	2.12	0.22	0.23
query25	0.16	0.09	0.08
query26	0.26	0.18	0.18
query27	0.07	0.08	0.07
query28	13.21	1.02	1.00
query29	12.60	3.34	3.33
query30	0.26	0.07	0.06
query31	2.86	0.40	0.39
query32	3.25	0.47	0.48
query33	2.86	2.85	2.92
query34	16.94	4.47	4.41
query35	4.55	4.48	4.46
query36	0.65	0.46	0.45
query37	0.18	0.15	0.15
query38	0.16	0.15	0.14
query39	0.04	0.03	0.04
query40	0.18	0.14	0.14
query41	0.09	0.06	0.04
query42	0.06	0.05	0.04
query43	0.05	0.04	0.04
Total cold run time: 109.65 s
Total hot run time: 30.25 s

…ls (#36431) bp: #36432 ## Proposed changes ## Fixed Bugs introduced from #34307 1. `FileSystemCacheKey.equals()` compares properties by `==`, resulting in creating new file system in each partition 2. `dfsFileSystem` is not synchronized, resulting in creating more file systems than need. 3. `jobConf.iterator()` will produce more than 2000 pairs of key-value

github-actions · 2024-06-20T07:00:18Z

PR approved by at least one committer and no changes requested.

kaka11chen

LGTM

…ls (#36432) ## Proposed changes ## Fixed Bugs introduced from #33937 1. `FileSystemCacheKey.equals()` compares properties by `==`, resulting in creating new file system in each partition 2. `dfsFileSystem` is not synchronized, resulting in creating more file systems than need. 3. `jobConf.iterator()` will produce more than 2000 pairs of key-value

…ls (apache#36432) ## Proposed changes ## Fixed Bugs introduced from apache#33937 1. `FileSystemCacheKey.equals()` compares properties by `==`, resulting in creating new file system in each partition 2. `dfsFileSystem` is not synchronized, resulting in creating more file systems than need. 3. `jobConf.iterator()` will produce more than 2000 pairs of key-value

AshinGau mentioned this pull request Jun 18, 2024
[fix](split) FileSystemCacheKey are always different in overload equals #36431
Merged

AshinGau force-pushed the fs_cache_key branch from aba8fd3 to 49887fbCompare June 18, 2024 01:42

morningman previously approved these changes Jun 19, 2024
View reviewed changes

github-actionsBot added the approved Indicates a PR has been approved by one committer. label Jun 19, 2024

github-actionsBot added the reviewed label Jun 19, 2024

[fix](split) FileSystemCacheKey are always different in overload equals
3558731
This reverts commit efe8e4c.

AshinGau dismissed morningman’s stale review via 3558731June 19, 2024 08:31

AshinGau force-pushed the fs_cache_key branch from 55c5960 to 3558731Compare June 19, 2024 08:31

github-actionsBot removed the approved Indicates a PR has been approved by one committer. label Jun 19, 2024

morningman approved these changes Jun 20, 2024
View reviewed changes

github-actionsBot added the approved Indicates a PR has been approved by one committer. label Jun 20, 2024

morningman added the dev/2.1.4-merged label Jun 20, 2024

kaka11chen approved these changes Jun 20, 2024
View reviewed changes

AshinGau merged commit ed89565 into apache:masterJun 21, 2024

dataroaring added the dev/3.0.0-merged label Aug 5, 2024

Uh oh!

Conversation

AshinGau commented Jun 18, 2024• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Proposed changes

Fixed Bugs introduced from #33937

Uh oh!

doris-robot commented Jun 18, 2024

Uh oh!

AshinGau commented Jun 18, 2024

Uh oh!

AshinGau commented Jun 18, 2024

Uh oh!

doris-robot commented Jun 19, 2024

Uh oh!

doris-robot commented Jun 19, 2024

Uh oh!

doris-robot commented Jun 19, 2024

Uh oh!

morningman left a comment

Choose a reason for hiding this comment

Uh oh!

github-actionsBot commented Jun 19, 2024

Uh oh!

github-actionsBot commented Jun 19, 2024

Uh oh!

AshinGau commented Jun 19, 2024

Uh oh!

doris-robot commented Jun 19, 2024

Uh oh!

doris-robot commented Jun 19, 2024

Uh oh!

doris-robot commented Jun 19, 2024

Uh oh!

github-actionsBot commented Jun 20, 2024

Uh oh!

kaka11chen left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

AshinGau commented Jun 18, 2024•
edited
Loading