[bugfix](hive)Delete the temporarily created folder by wuwenchi · Pull Request #40424 · apache/doris

wuwenchi · 2024-09-05T07:54:03Z

Proposed changes

Delete the temporarily created folder, otherwise it will cause too many folders on hdfs:

The directory item limit of xxx is exceeded: limit=xxx items=xxx

doris-robot · 2024-09-05T07:54:08Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

wuwenchi · 2024-09-05T07:54:51Z

run buildall

doris-robot · 2024-09-05T09:07:25Z

TPC-H: Total hot run time: 38333 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit e00c785e9e9a8ac64bb8fcb9f7eb4a661891ab70, data reload: false
------ Round 1 ----------------------------------
q1	17600	4571	4325	4325
q2	2029	195	191	191
q3	11733	972	1083	972
q4	10522	702	695	695
q5	7772	2908	2792	2792
q6	231	139	139	139
q7	965	617	621	617
q8	9328	2107	2068	2068
q9	7080	6583	6577	6577
q10	7008	2240	2288	2240
q11	471	252	240	240
q12	409	229	230	229
q13	17761	3074	3071	3071
q14	290	245	241	241
q15	541	500	484	484
q16	599	500	519	500
q17	1006	678	684	678
q18	7421	6829	6807	6807
q19	1396	1074	1060	1060
q20	693	346	330	330
q21	4670	3169	3075	3075
q22	1098	1017	1002	1002
Total cold run time: 110623 ms
Total hot run time: 38333 ms
----- Round 2, with runtime_filter_mode=off -----
q1	4365	4280	4275	4275
q2	386	271	278	271
q3	2926	2685	2657	2657
q4	1935	1693	1696	1693
q5	5700	5725	5841	5725
q6	238	130	134	130
q7	2209	1894	1847	1847
q8	3283	3469	3519	3469
q9	8848	8831	8840	8831
q10	3685	3462	3301	3301
q11	616	511	497	497
q12	850	707	682	682
q13	14915	3224	3296	3224
q14	310	305	282	282
q15	549	509	521	509
q16	587	600	599	599
q17	1841	1597	1554	1554
q18	8186	7868	7829	7829
q19	1764	1633	1495	1495
q20	2134	1964	1922	1922
q21	5623	5633	5440	5440
q22	1135	1052	1071	1052
Total cold run time: 72085 ms
Total hot run time: 57284 ms

wuwenchi · 2024-09-05T09:12:02Z

run feut

doris-robot · 2024-09-05T09:19:34Z

TPC-DS: Total hot run time: 192803 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit e00c785e9e9a8ac64bb8fcb9f7eb4a661891ab70, data reload: false
query1	1257	916	855	855
query2	6445	1860	1857	1857
query3	10611	4068	4002	4002
query4	59329	23829	23377	23377
query5	5319	498	500	498
query6	408	172	157	157
query7	5770	305	288	288
query8	312	215	222	215
query9	8957	2510	2499	2499
query10	489	273	251	251
query11	17404	15088	15174	15088
query12	154	104	113	104
query13	1574	397	386	386
query14	11115	7606	7319	7319
query15	223	179	181	179
query16	7214	455	469	455
query17	1110	572	568	568
query18	1917	294	314	294
query19	296	153	157	153
query20	129	113	109	109
query21	214	113	103	103
query22	4592	4457	4476	4457
query23	34424	33296	33562	33296
query24	5921	2931	2808	2808
query25	519	391	392	391
query26	673	153	152	152
query27	1773	276	282	276
query28	3527	2081	2054	2054
query29	646	405	398	398
query30	235	156	155	155
query31	938	757	737	737
query32	86	52	51	51
query33	446	279	276	276
query34	879	476	487	476
query35	828	726	738	726
query36	1075	948	958	948
query37	176	90	94	90
query38	4106	3893	3904	3893
query39	1463	1392	1419	1392
query40	201	113	112	112
query41	47	46	44	44
query42	117	94	94	94
query43	515	472	469	469
query44	1081	759	741	741
query45	194	170	164	164
query46	1090	735	770	735
query47	1856	1785	1814	1785
query48	378	305	303	303
query49	766	434	432	432
query50	822	425	418	418
query51	6999	6888	6824	6824
query52	94	85	86	85
query53	249	178	181	178
query54	566	468	461	461
query55	77	75	73	73
query56	265	278	256	256
query57	1181	1078	1067	1067
query58	224	238	234	234
query59	3010	2887	2825	2825
query60	297	268	276	268
query61	104	99	102	99
query62	784	661	664	661
query63	214	183	180	180
query64	2579	709	786	709
query65	3223	3145	3145	3145
query66	686	337	337	337
query67	15475	15089	15057	15057
query68	2998	589	581	581
query69	397	289	278	278
query70	1090	1153	1040	1040
query71	366	279	278	278
query72	5352	4098	3988	3988
query73	745	329	331	329
query74	9181	8922	8777	8777
query75	3366	2693	2684	2684
query76	1448	981	978	978
query77	514	310	307	307
query78	10752	9899	9166	9166
query79	1067	548	538	538
query80	706	491	527	491
query81	517	239	241	239
query82	254	150	145	145
query83	169	153	149	149
query84	262	75	80	75
query85	785	292	275	275
query86	300	302	294	294
query87	4426	4336	4260	4260
query88	3123	2395	2388	2388
query89	373	286	284	284
query90	1960	187	186	186
query91	122	97	97	97
query92	56	50	50	50
query93	1035	532	535	532
query94	696	288	285	285
query95	310	248	251	248
query96	600	266	272	266
query97	3150	3101	3032	3032
query98	234	205	217	205
query99	1553	1302	1269	1269
Total cold run time: 303555 ms
Total hot run time: 192803 ms

doris-robot · 2024-09-05T09:25:09Z

ClickBench: Total hot run time: 31.95 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit e00c785e9e9a8ac64bb8fcb9f7eb4a661891ab70, data reload: false
query1	0.04	0.04	0.04
query2	0.08	0.04	0.04
query3	0.22	0.05	0.05
query4	1.67	0.07	0.08
query5	0.54	0.49	0.48
query6	1.13	0.73	0.74
query7	0.02	0.01	0.02
query8	0.05	0.05	0.05
query9	0.56	0.51	0.49
query10	0.54	0.57	0.53
query11	0.16	0.11	0.12
query12	0.14	0.12	0.12
query13	0.60	0.58	0.60
query14	2.11	2.07	2.03
query15	0.85	0.83	0.81
query16	0.37	0.37	0.38
query17	0.97	1.01	1.06
query18	0.20	0.19	0.19
query19	1.87	1.84	1.82
query20	0.01	0.00	0.01
query21	15.38	0.68	0.66
query22	4.91	6.54	1.96
query23	18.27	1.45	1.26
query24	2.24	0.22	0.22
query25	0.15	0.08	0.09
query26	0.27	0.18	0.17
query27	0.08	0.08	0.08
query28	13.18	1.01	0.99
query29	12.60	3.34	3.27
query30	0.24	0.05	0.06
query31	2.87	0.39	0.39
query32	3.27	0.50	0.47
query33	2.96	2.99	3.11
query34	16.86	4.37	4.32
query35	4.51	4.44	4.46
query36	0.65	0.47	0.47
query37	0.19	0.16	0.16
query38	0.16	0.15	0.15
query39	0.05	0.03	0.04
query40	0.16	0.14	0.13
query41	0.09	0.05	0.05
query42	0.06	0.05	0.04
query43	0.05	0.04	0.04
Total cold run time: 111.33 s
Total hot run time: 31.95 s

morningman

LGTM

github-actions · 2024-09-05T14:30:23Z

PR approved by at least one committer and no changes requested.

github-actions · 2024-09-05T14:30:25Z

PR approved by anyone and no changes requested.

morningman · 2024-09-06T08:28:46Z

run buildall

doris-robot · 2024-09-06T09:45:07Z

TPC-H: Total hot run time: 38090 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit d5dc7aa4dc2e7491effdd32ce0a5510d916a7cc2, data reload: false
------ Round 1 ----------------------------------
q1	18444	4542	4370	4370
q2	2021	193	181	181
q3	10467	1192	1042	1042
q4	10138	716	708	708
q5	7740	2898	2816	2816
q6	224	138	138	138
q7	961	622	614	614
q8	9361	2078	2043	2043
q9	7296	6554	6483	6483
q10	7028	2192	2318	2192
q11	469	251	251	251
q12	404	223	221	221
q13	17776	3090	3068	3068
q14	289	257	237	237
q15	537	486	482	482
q16	497	426	434	426
q17	995	677	732	677
q18	7444	6918	6851	6851
q19	1394	1013	1015	1013
q20	691	345	327	327
q21	4079	3133	2966	2966
q22	1097	996	984	984
Total cold run time: 109352 ms
Total hot run time: 38090 ms
----- Round 2, with runtime_filter_mode=off -----
q1	4350	4320	4330	4320
q2	376	276	273	273
q3	2898	2615	2669	2615
q4	1966	1675	1671	1671
q5	5396	5403	5452	5403
q6	218	133	132	132
q7	2119	1729	1779	1729
q8	3206	3382	3352	3352
q9	8385	8400	8443	8400
q10	3437	3204	3190	3190
q11	596	525	514	514
q12	782	597	620	597
q13	10947	3108	3076	3076
q14	290	281	262	262
q15	521	494	478	478
q16	521	467	487	467
q17	1827	1522	1478	1478
q18	7842	7327	7426	7327
q19	1675	1539	1483	1483
q20	2057	1807	1821	1807
q21	5485	5426	5201	5201
q22	1112	1033	999	999
Total cold run time: 66006 ms
Total hot run time: 54774 ms

doris-robot · 2024-09-06T09:56:52Z

TPC-DS: Total hot run time: 187669 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit d5dc7aa4dc2e7491effdd32ce0a5510d916a7cc2, data reload: false
query1	921	383	383	383
query2	6507	1962	1910	1910
query3	6652	206	215	206
query4	29123	23321	23123	23123
query5	4148	497	501	497
query6	267	175	169	169
query7	4593	298	297	297
query8	293	220	219	219
query9	8590	2501	2470	2470
query10	434	274	268	268
query11	17859	14978	15121	14978
query12	157	104	109	104
query13	1642	399	368	368
query14	9867	7181	6912	6912
query15	255	173	176	173
query16	8064	481	464	464
query17	1593	577	567	567
query18	2136	295	294	294
query19	226	151	152	151
query20	122	114	112	112
query21	210	110	104	104
query22	4399	4217	4264	4217
query23	34030	33246	33275	33246
query24	11241	2924	2868	2868
query25	650	420	419	419
query26	1481	161	156	156
query27	2875	281	279	279
query28	7649	2046	2020	2020
query29	958	441	427	427
query30	317	155	152	152
query31	982	780	795	780
query32	98	62	57	57
query33	778	296	297	296
query34	967	493	496	493
query35	862	725	751	725
query36	1102	945	967	945
query37	160	87	86	86
query38	3996	3877	3751	3751
query39	1443	1388	1389	1388
query40	272	123	120	120
query41	52	50	49	49
query42	120	99	99	99
query43	513	465	471	465
query44	1246	759	748	748
query45	201	176	172	172
query46	1103	733	746	733
query47	1869	1780	1819	1780
query48	371	292	300	292
query49	1213	464	460	460
query50	837	417	417	417
query51	7033	6836	6852	6836
query52	100	92	90	90
query53	259	190	184	184
query54	1068	470	458	458
query55	84	77	75	75
query56	285	272	276	272
query57	1167	1064	1095	1064
query58	254	236	241	236
query59	3051	2889	2896	2889
query60	304	280	303	280
query61	127	125	122	122
query62	851	638	698	638
query63	230	195	198	195
query64	5298	700	650	650
query65	3215	3152	3173	3152
query66	1441	372	336	336
query67	15336	15337	15281	15281
query68	4785	560	571	560
query69	423	280	296	280
query70	1182	1112	1128	1112
query71	421	270	278	270
query72	6763	3983	4007	3983
query73	759	327	326	326
query74	9066	8773	8871	8773
query75	3528	2686	2680	2680
query76	2769	1012	999	999
query77	548	325	317	317
query78	9657	8967	9005	8967
query79	1417	558	553	553
query80	777	529	515	515
query81	583	235	228	228
query82	560	140	138	138
query83	271	194	155	155
query84	235	82	77	77
query85	1422	289	278	278
query86	429	295	258	258
query87	4389	4141	4176	4141
query88	3998	2298	2294	2294
query89	383	293	288	288
query90	1848	195	209	195
query91	130	103	100	100
query92	66	52	53	52
query93	1968	560	558	558
query94	884	303	293	293
query95	351	257	254	254
query96	593	309	272	272
query97	3209	3124	3103	3103
query98	211	203	202	202
query99	1703	1281	1283	1281
Total cold run time: 290875 ms
Total hot run time: 187669 ms

doris-robot · 2024-09-06T10:02:25Z

ClickBench: Total hot run time: 31.43 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit d5dc7aa4dc2e7491effdd32ce0a5510d916a7cc2, data reload: false
query1	0.05	0.04	0.04
query2	0.08	0.04	0.04
query3	0.23	0.05	0.05
query4	1.67	0.08	0.07
query5	0.49	0.48	0.50
query6	1.15	0.73	0.72
query7	0.02	0.01	0.01
query8	0.05	0.05	0.05
query9	0.54	0.50	0.49
query10	0.55	0.59	0.55
query11	0.16	0.12	0.12
query12	0.15	0.12	0.12
query13	0.60	0.58	0.58
query14	1.36	1.41	1.40
query15	0.84	0.80	0.80
query16	0.36	0.37	0.37
query17	0.96	1.00	1.03
query18	0.21	0.19	0.20
query19	1.84	1.76	1.79
query20	0.01	0.01	0.01
query21	15.40	0.65	0.65
query22	4.12	6.75	2.05
query23	18.25	1.37	1.25
query24	2.14	0.22	0.22
query25	0.16	0.08	0.09
query26	0.28	0.18	0.18
query27	0.08	0.08	0.08
query28	13.20	1.03	1.00
query29	12.59	3.33	3.29
query30	0.25	0.05	0.06
query31	2.89	0.40	0.39
query32	3.26	0.49	0.47
query33	3.01	3.01	2.98
query34	16.96	4.37	4.39
query35	4.49	4.42	4.42
query36	0.66	0.49	0.50
query37	0.19	0.16	0.16
query38	0.16	0.15	0.15
query39	0.05	0.04	0.04
query40	0.16	0.12	0.12
query41	0.10	0.05	0.05
query42	0.06	0.06	0.05
query43	0.05	0.04	0.04
Total cold run time: 109.83 s
Total hot run time: 31.43 s

wuwenchi · 2024-09-09T02:09:28Z

run feut

morningman

LGTM

kaka11chen

LGTM

## Proposed changes Delete the temporarily created folder, otherwise it will cause too many folders on hdfs: > The directory item limit of xxx is exceeded: limit=xxx items=xxx

## Proposed changes bp: #40424

## Proposed changes Delete the temporarily created folder, otherwise it will cause too many folders on hdfs: > The directory item limit of xxx is exceeded: limit=xxx items=xxx

morningman added dev/2.1.x dev/3.0.x labels Sep 5, 2024

morningman approved these changes Sep 5, 2024
View reviewed changes

github-actionsBot added the approved Indicates a PR has been approved by one committer. label Sep 5, 2024

github-actionsBot added the reviewed label Sep 5, 2024

fix hive
d5dc7aa

morningman force-pushed the fix-hive-delete-dir branch from e00c785 to d5dc7aaCompare September 6, 2024 08:28

morningman approved these changes Sep 10, 2024
View reviewed changes

kaka11chen approved these changes Sep 10, 2024
View reviewed changes

morningman merged commit c7f57ae into apache:masterSep 10, 2024

wuwenchi mentioned this pull request Sep 12, 2024
[bugfix](hive)Delete the temporarily created folder (#40424) #40722
Merged

yiguolei pushed a commit that referenced this pull request Sep 12, 2024
[bugfix](hive)Delete the temporarily created folder (#40424) (#40722)
3d2ac60
## Proposed changes bp: #40424

yiguolei added dev/2.1.7-merged and removed dev/2.1.x labels Sep 12, 2024

morningman removed the dev/3.0.x label Oct 9, 2024

morningman added the dev/3.0.3-merged label Oct 9, 2024

yiguolei mentioned this pull request Nov 6, 2024
Release Note 2.1.7 #43319
Closed

yiguolei mentioned this pull request Jan 19, 2025
Release Note 2.1.8 #47198
Closed

Uh oh!

Conversation

wuwenchi commented Sep 5, 2024

Proposed changes

Uh oh!

doris-robot commented Sep 5, 2024

Uh oh!

wuwenchi commented Sep 5, 2024

Uh oh!

doris-robot commented Sep 5, 2024

Uh oh!

wuwenchi commented Sep 5, 2024

Uh oh!

doris-robot commented Sep 5, 2024

Uh oh!

doris-robot commented Sep 5, 2024

Uh oh!

morningman left a comment

Choose a reason for hiding this comment

Uh oh!

github-actionsBot commented Sep 5, 2024

Uh oh!

github-actionsBot commented Sep 5, 2024

Uh oh!

morningman commented Sep 6, 2024

Uh oh!

doris-robot commented Sep 6, 2024

Uh oh!

doris-robot commented Sep 6, 2024

Uh oh!

doris-robot commented Sep 6, 2024

Uh oh!

wuwenchi commented Sep 9, 2024

Uh oh!

morningman left a comment

Choose a reason for hiding this comment

Uh oh!

kaka11chen left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants