Skip to content

[Fix](outfile) Fix the timing of setting the _is_closed flag in Parquet/ORC writer - #34668

Merged
morningman merged 1 commit into
apache:masterfrom
BePPPower:fixParquetClose
May 14, 2024
Merged

[Fix](outfile) Fix the timing of setting the _is_closed flag in Parquet/ORC writer#34668
morningman merged 1 commit into
apache:masterfrom
BePPPower:fixParquetClose

Conversation

@BePPPower

@BePPPowerBePPPower commented May 10, 2024•
edited
Loading

Copy link
Copy Markdown
Contributor

Proposed changes

Issue Number: close #xxx

In parquet_transformer.cpp:
If _file_writer->close() returns an error, then _is_closed = true will not be executed.

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

@doris-robot

doris-robot commented May 10, 2024

Copy link
Copy Markdown

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@BePPPower

BePPPower commented May 10, 2024

Copy link
Copy Markdown
ContributorAuthor

run buildall

@BePPPowerBePPPower mentioned this pull request May 10, 2024
Merged
@doris-robot

doris-robot commented May 10, 2024

Copy link
Copy Markdown

TeamCity be ut coverage result:
Function Coverage: 35.68% (8981/25174)
Line Coverage: 27.34% (74251/271562)
Region Coverage: 26.58% (38384/144430)
Branch Coverage: 23.40% (19576/83658)
Coverage Report: http://coverage.selectdb-in.cc/coverage/38e22336e569a81a757b7bf9595bf076c4e717cd_38e22336e569a81a757b7bf9595bf076c4e717cd/report/index.html

morningman
morningman approved these changes May 12, 2024
View reviewed changes

@morningmanmorningman left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@morningman

morningman commented May 12, 2024

Copy link
Copy Markdown
Contributor

run buildall

@github-actionsgithub-actionsBot added the approved Indicates a PR has been approved by one committer. label May 12, 2024
@github-actions

github-actionsBot commented May 12, 2024

Copy link
Copy Markdown
Contributor

PR approved by at least one committer and no changes requested.

@github-actions

github-actionsBot commented May 12, 2024

Copy link
Copy Markdown
Contributor

PR approved by anyone and no changes requested.

@morningmanmorningman added dev/2.0.x and removed approved Indicates a PR has been approved by one committer. reviewed labels May 12, 2024
@github-actions

github-actionsBot commented May 12, 2024

Copy link
Copy Markdown
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@doris-robot

doris-robot commented May 12, 2024

Copy link
Copy Markdown

TeamCity be ut coverage result:
Function Coverage: 35.65% (8984/25203)
Line Coverage: 27.32% (74277/271866)
Region Coverage: 26.56% (38392/144574)
Branch Coverage: 23.37% (19578/83766)
Coverage Report: http://coverage.selectdb-in.cc/coverage/eaee586a17b95fbe2fd63b99e5cfda62b000dd09_eaee586a17b95fbe2fd63b99e5cfda62b000dd09/report/index.html

@doris-robot

doris-robot commented May 12, 2024

Copy link
Copy Markdown
TPC-H: Total hot run time: 41877 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit eaee586a17b95fbe2fd63b99e5cfda62b000dd09, data reload: false
------ Round 1 ----------------------------------
q1	17642	4265	4203	4203
q2	2021	187	189	187
q3	10442	1260	1239	1239
q4	10205	777	841	777
q5	7459	2671	2763	2671
q6	224	136	131	131
q7	1006	594	585	585
q8	9228	2113	2094	2094
q9	9271	6706	6716	6706
q10	9031	3803	3849	3803
q11	466	242	240	240
q12	482	220	220	220
q13	17388	3198	3220	3198
q14	279	234	225	225
q15	519	468	489	468
q16	515	432	414	414
q17	980	702	654	654
q18	8223	7817	7781	7781
q19	6696	1574	1532	1532
q20	655	320	331	320
q21	5280	4147	4223	4147
q22	353	288	282	282
Total cold run time: 118365 ms
Total hot run time: 41877 ms
----- Round 2, with runtime_filter_mode=off -----
q1	4583	4338	4408	4338
q2	368	280	268	268
q3	3182	2965	2966	2965
q4	2022	1620	1642	1620
q5	5443	5500	5537	5500
q6	213	123	122	122
q7	2345	1949	1917	1917
q8	3271	3396	3413	3396
q9	8578	8717	8647	8647
q10	3940	3698	3789	3698
q11	589	521	485	485
q12	791	619	623	619
q13	17023	3135	3124	3124
q14	285	262	277	262
q15	524	483	483	483
q16	482	456	448	448
q17	1826	1509	1476	1476
q18	7686	7513	7409	7409
q19	1652	1487	1537	1487
q20	2005	1765	1781	1765
q21	12840	4802	4824	4802
q22	580	508	502	502
Total cold run time: 80228 ms
Total hot run time: 55333 ms

@doris-robot

doris-robot commented May 12, 2024

Copy link
Copy Markdown
TPC-DS: Total hot run time: 186978 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit eaee586a17b95fbe2fd63b99e5cfda62b000dd09, data reload: false
query1	919	360	353	353
query2	7166	2401	2469	2401
query3	6639	213	212	212
query4	22815	21165	21145	21145
query5	4135	411	413	411
query6	276	175	171	171
query7	4582	296	285	285
query8	238	197	199	197
query9	8523	2439	2421	2421
query10	433	265	264	264
query11	14841	14258	14234	14234
query12	133	100	97	97
query13	1655	379	372	372
query14	10706	7533	7581	7533
query15	212	170	171	170
query16	7690	267	256	256
query17	1306	555	539	539
query18	1939	269	290	269
query19	194	145	145	145
query20	87	85	84	84
query21	190	133	127	127
query22	4992	4850	4801	4801
query23	34078	33686	33618	33618
query24	8819	2860	2964	2860
query25	590	380	363	363
query26	702	156	152	152
query27	2244	319	314	314
query28	5766	2080	2058	2058
query29	843	601	614	601
query30	268	148	153	148
query31	949	768	740	740
query32	95	56	54	54
query33	583	250	243	243
query34	861	494	480	480
query35	771	678	700	678
query36	1066	890	945	890
query37	105	67	67	67
query38	2898	2819	2768	2768
query39	1619	1557	1548	1548
query40	205	130	124	124
query41	42	62	37	37
query42	104	95	94	94
query43	601	562	573	562
query44	1057	724	739	724
query45	267	248	254	248
query46	1067	707	744	707
query47	1955	1864	1908	1864
query48	379	303	300	300
query49	839	397	394	394
query50	769	386	397	386
query51	6976	6819	6764	6764
query52	107	89	90	89
query53	354	279	276	276
query54	636	433	422	422
query55	72	70	73	70
query56	235	220	219	219
query57	1250	1152	1157	1152
query58	218	200	206	200
query59	3526	3209	3151	3151
query60	254	233	234	233
query61	126	90	91	90
query62	616	471	474	471
query63	311	280	283	280
query64	8478	7459	7324	7324
query65	3102	3089	3065	3065
query66	784	350	347	347
query67	15505	15035	14815	14815
query68	4509	538	565	538
query69	482	320	308	308
query70	1170	1107	1146	1107
query71	384	273	325	273
query72	7069	2610	2347	2347
query73	700	328	326	326
query74	6496	6123	6069	6069
query75	3307	2639	2615	2615
query76	2235	986	997	986
query77	389	263	261	261
query78	10646	10035	10046	10035
query79	2460	515	504	504
query80	1090	452	429	429
query81	544	218	220	218
query82	926	96	94	94
query83	237	165	162	162
query84	246	87	82	82
query85	1268	271	318	271
query86	457	293	301	293
query87	3312	3195	3125	3125
query88	4414	2454	2469	2454
query89	483	398	399	398
query90	1996	195	187	187
query91	123	99	98	98
query92	54	48	50	48
query93	1853	510	489	489
query94	1296	182	181	181
query95	402	293	303	293
query96	594	269	270	269
query97	3145	2949	2976	2949
query98	235	211	221	211
query99	1174	881	904	881
Total cold run time: 275459 ms
Total hot run time: 186978 ms

@github-actionsgithub-actionsBot added the approved Indicates a PR has been approved by one committer. label May 14, 2024
@github-actions

github-actionsBot commented May 14, 2024

Copy link
Copy Markdown
Contributor

PR approved by at least one committer and no changes requested.

@github-actions

github-actionsBot commented May 14, 2024

Copy link
Copy Markdown
Contributor

PR approved by anyone and no changes requested.

@morningman
morningman merged commit 3b00ba0 into apache:masterMay 14, 2024
yiguolei pushed a commit that referenced this pull request May 15, 2024
@BePPPower@Doris-Extras
[Fix](outfile) Fix the timing of setting the _is_closed flag in Parqu…
c7134fa
…et/ORC writer (#34668)
morningman pushed a commit that referenced this pull request May 16, 2024
@BePPPower
[Fix](outfile) Fix the timing of setting the _is_closed flag in Parqu…
ccf8be8
…et/ORC writer (#34669)
backport: #34668
mongo360 pushed a commit to mongo360/doris that referenced this pull request Aug 16, 2024
@BePPPowerweixingyu12
[Fix](outfile) Fix the timing of setting the _is_closed flag in Parqu…
01534c2
…et/ORC writer (apache#34669)
backport: apache#34668
HappenLee pushed a commit to HappenLee/incubator-doris that referenced this pull request Apr 24, 2026
@BePPPower@HappenLee
[Fix](outfile) Fix the timing of setting the _is_closed flag in Parqu…
fb41287
…et/ORC writer (apache#34668)
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@morningmanmorningmanmorningman approved these changes
@zddrzddrzddr approved these changes

Assignees

No one assigned

Labels

approvedIndicates a PR has been approved by one committer.dev/2.0.11-mergeddev/2.1.3-mergeddev/3.0.0-mergedreviewed

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@BePPPower@doris-robot@morningman@zddr@gavinchou