[fix](outfile) fix small file output with bz2 compression by morningman · Pull Request #56368 · apache/doris

morningman · 2025-09-23T20:44:42Z

What problem does this PR solve?

When export small data with bz2 compression, it may return BZ_OUTBUFF_FULL,
because the pre allocated output buffer is too small.
This PR fix this by adding a fix size to avoid this problem

Release note

None

Check List (For Author)

Test
- Regression test
- Unit Test
- Manual test (add detailed scripts or steps below)
- No need to test or manual test. Explain why:
  - This is a refactor/code format and no logic has been changed.
  - Previous test can cover this change.
  - No code files have been changed.
  - Other reason
Behavior changed:
- No.
- Yes.
Does this need documentation?
- No.
- Yes.

Check List (For Reviewer who merge this PR)

Confirm the release note
Confirm test cases
Confirm document
Add branch pick label

hello-stephen · 2025-09-23T20:44:48Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

What problem was fixed (it's best to include specific error reporting information). How it was fixed.
Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
What features were added. Why was this function added?
Which code was refactored and why was this part of the code refactored?
Which functions were optimized and what is the difference before and after the optimization?

morningman · 2025-09-23T21:11:30Z

run buildall

doris-robot · 2025-09-23T22:01:59Z

ClickBench: Total hot run time: 30.72 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 4e3e3eadbe14faf436560b3458661b3006aa5312, data reload: false
query1	0.06	0.05	0.05
query2	0.10	0.05	0.06
query3	0.26	0.09	0.09
query4	1.61	0.11	0.11
query5	0.29	0.27	0.25
query6	1.14	0.67	0.64
query7	0.04	0.03	0.03
query8	0.05	0.04	0.04
query9	0.60	0.53	0.52
query10	0.57	0.58	0.58
query11	0.16	0.12	0.12
query12	0.15	0.12	0.16
query13	0.63	0.63	0.62
query14	1.05	1.06	1.03
query15	0.87	0.87	0.87
query16	0.40	0.41	0.41
query17	1.06	1.07	1.08
query18	0.21	0.21	0.21
query19	2.24	2.04	2.02
query20	0.02	0.02	0.01
query21	15.63	0.96	0.60
query22	0.75	1.10	0.71
query23	14.93	1.44	0.68
query24	6.82	1.76	0.59
query25	0.52	0.27	0.07
query26	0.63	0.16	0.15
query27	0.06	0.06	0.06
query28	9.28	1.37	0.94
query29	12.62	4.03	3.32
query30	0.30	0.13	0.11
query31	2.82	0.60	0.38
query32	3.25	0.60	0.48
query33	3.12	3.15	3.18
query34	16.07	5.55	4.88
query35	4.97	4.92	4.90
query36	0.68	0.53	0.49
query37	0.11	0.07	0.07
query38	0.06	0.06	0.04
query39	0.04	0.03	0.02
query40	0.18	0.17	0.15
query41	0.09	0.04	0.03
query42	0.04	0.04	0.03
query43	0.05	0.04	0.03
Total cold run time: 104.53 s
Total hot run time: 30.72 s

hello-stephen · 2025-09-23T22:26:38Z

BE UT Coverage Report

Increment line coverage 0.00% (0/1) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	52.41% (17641/33661)
Line Coverage	37.64% (160183/425571)
Region Coverage	32.15% (121949/379255)
Branch Coverage	33.51% (53473/159590)

hello-stephen · 2025-09-23T23:59:25Z

BE Regression && UT Coverage Report

Increment line coverage 100.00% (1/1) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	71.12% (23382/32879)
Line Coverage	57.53% (244176/424465)
Region Coverage	52.84% (203077/384307)
Branch Coverage	54.50% (87340/160243)

morningman · 2025-09-24T21:18:37Z

run buildall

doris-robot · 2025-09-24T22:08:03Z

ClickBench: Total hot run time: 30.6 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit df7fcd317a28500e6e559930a5f71ea44cef5e91, data reload: false
query1	0.06	0.04	0.04
query2	0.10	0.05	0.05
query3	0.26	0.09	0.08
query4	1.61	0.11	0.12
query5	0.27	0.26	0.25
query6	1.19	0.64	0.65
query7	0.04	0.03	0.03
query8	0.05	0.04	0.05
query9	0.61	0.54	0.52
query10	0.57	0.58	0.57
query11	0.17	0.12	0.11
query12	0.15	0.12	0.12
query13	0.63	0.63	0.62
query14	1.04	1.06	1.03
query15	0.87	0.85	0.86
query16	0.39	0.40	0.38
query17	1.03	1.06	1.07
query18	0.22	0.20	0.20
query19	2.00	1.87	1.83
query20	0.02	0.01	0.01
query21	15.42	0.94	0.58
query22	0.75	1.18	0.69
query23	15.01	1.39	0.64
query24	6.78	0.94	1.41
query25	0.46	0.20	0.09
query26	0.60	0.16	0.14
query27	0.08	0.06	0.06
query28	9.73	1.37	0.93
query29	12.59	3.85	3.24
query30	0.29	0.14	0.12
query31	2.83	0.59	0.39
query32	3.24	0.56	0.47
query33	3.12	3.22	3.04
query34	16.09	5.51	4.87
query35	4.99	4.97	4.98
query36	0.71	0.52	0.52
query37	0.11	0.08	0.07
query38	0.07	0.05	0.05
query39	0.04	0.03	0.02
query40	0.18	0.15	0.14
query41	0.08	0.03	0.04
query42	0.04	0.04	0.03
query43	0.05	0.04	0.03
Total cold run time: 104.54 s
Total hot run time: 30.6 s

doris-robot · 2025-09-24T22:36:41Z

BE UT Coverage Report

Increment line coverage 0.00% (0/1) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	52.40% (17609/33604)
Line Coverage	37.62% (159960/425244)
Region Coverage	32.15% (121965/379325)
Branch Coverage	33.51% (53495/159656)

morningman · 2025-09-25T01:52:42Z

run buildall

doris-robot · 2025-09-25T02:49:42Z

ClickBench: Total hot run time: 29.96 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 04d746711764283b1cbafe40b1f849e10a5bcb6e, data reload: false
query1	0.06	0.05	0.05
query2	0.08	0.05	0.05
query3	0.25	0.09	0.09
query4	1.61	0.12	0.12
query5	0.28	0.27	0.25
query6	1.21	0.63	0.65
query7	0.03	0.02	0.02
query8	0.05	0.05	0.05
query9	0.63	0.52	0.52
query10	0.61	0.57	0.59
query11	0.17	0.11	0.11
query12	0.15	0.12	0.13
query13	0.64	0.63	0.62
query14	1.02	1.02	1.01
query15	0.88	0.84	0.89
query16	0.41	0.39	0.41
query17	1.05	1.08	1.09
query18	0.22	0.20	0.20
query19	1.97	1.78	1.89
query20	0.02	0.02	0.02
query21	15.42	0.92	0.56
query22	0.77	1.27	0.68
query23	14.81	1.36	0.65
query24	7.12	1.17	0.36
query25	0.38	0.19	0.07
query26	0.66	0.18	0.13
query27	0.07	0.06	0.06
query28	8.75	1.38	0.94
query29	12.58	3.91	3.26
query30	0.28	0.14	0.11
query31	2.84	0.60	0.39
query32	3.27	0.58	0.47
query33	3.11	3.04	3.17
query34	16.10	5.46	4.92
query35	4.92	4.99	4.91
query36	0.69	0.50	0.50
query37	0.11	0.07	0.07
query38	0.07	0.05	0.05
query39	0.03	0.03	0.04
query40	0.18	0.16	0.14
query41	0.09	0.04	0.04
query42	0.05	0.03	0.03
query43	0.05	0.04	0.03
Total cold run time: 103.69 s
Total hot run time: 29.96 s

hello-stephen · 2025-09-25T02:58:57Z

BE UT Coverage Report

Increment line coverage 0.00% (0/1) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	52.40% (17610/33606)
Line Coverage	37.62% (160007/425308)
Region Coverage	32.17% (122048/379370)
Branch Coverage	33.51% (53513/159684)

morningman · 2025-09-25T05:56:44Z

run buildall

doris-robot · 2025-09-25T06:31:26Z

ClickBench: Total hot run time: 30.74 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 8f7b08dee3dd9f24dde23757d034506fd96351af, data reload: false
query1	0.05	0.05	0.05
query2	0.09	0.04	0.07
query3	0.25	0.08	0.09
query4	1.60	0.12	0.12
query5	0.28	0.28	0.26
query6	1.20	0.65	0.67
query7	0.04	0.03	0.03
query8	0.06	0.05	0.05
query9	0.63	0.53	0.54
query10	0.59	0.58	0.59
query11	0.16	0.11	0.11
query12	0.15	0.12	0.12
query13	0.62	0.62	0.63
query14	1.03	1.03	1.02
query15	0.89	0.84	0.86
query16	0.42	0.39	0.40
query17	1.05	1.06	1.08
query18	0.21	0.20	0.20
query19	1.93	1.82	1.84
query20	0.02	0.02	0.01
query21	15.47	0.94	0.56
query22	0.77	1.28	0.77
query23	14.78	1.41	0.64
query24	6.86	1.56	1.10
query25	0.48	0.16	0.07
query26	0.53	0.17	0.14
query27	0.07	0.05	0.05
query28	9.84	1.39	0.94
query29	12.61	4.00	3.24
query30	0.29	0.13	0.13
query31	2.85	0.60	0.39
query32	3.23	0.56	0.48
query33	3.09	3.12	3.03
query34	16.17	5.48	4.82
query35	4.90	4.89	4.90
query36	0.68	0.52	0.50
query37	0.11	0.08	0.08
query38	0.07	0.05	0.05
query39	0.04	0.03	0.03
query40	0.18	0.15	0.15
query41	0.08	0.03	0.03
query42	0.05	0.03	0.03
query43	0.05	0.04	0.04
Total cold run time: 104.47 s
Total hot run time: 30.74 s

doris-robot · 2025-09-25T07:04:08Z

BE UT Coverage Report

Increment line coverage 0.00% (0/1) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	52.40% (17611/33607)
Line Coverage	37.62% (160023/425318)
Region Coverage	32.16% (122002/379372)
Branch Coverage	33.51% (53518/159686)

hello-stephen · 2025-09-25T08:26:11Z

BE Regression && UT Coverage Report

Increment line coverage 100.00% (1/1) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	71.13% (23338/32810)
Line Coverage	57.54% (244079/424162)
Region Coverage	52.96% (203586/384402)
Branch Coverage	54.55% (87469/160335)

github-actions · 2025-09-26T04:45:07Z

PR approved by at least one committer and no changes requested.

github-actions · 2025-09-26T04:45:10Z

PR approved by anyone and no changes requested.

### What problem does this PR solve? When export small data with bz2 compression, it may return BZ_OUTBUFF_FULL, because the pre allocated output buffer is too small. This PR fix this by adding a fix size to avoid this problem

…56368 (#56495) Cherry-picked from #56368 Co-authored-by: Mingyu Chen (Rayner) <morningman@163.com>

When export small data with bz2 compression, it may return BZ_OUTBUFF_FULL, because the pre allocated output buffer is too small. This PR fix this by adding a fix size to avoid this problem

…56368 (#57041) bp #56368

…) (apache#4824) closeapache#4791

morningman added dev/2.1.x dev/3.0.x dev/3.1.x labels Sep 23, 2025

morningman added the dev/4.0.x label Sep 24, 2025

morningman force-pushed the fix_small_bz2 branch from df7fcd3 to 04d7467Compare September 25, 2025 01:52

morningman added 4 commits September 24, 2025 22:55

[fix](outfile) fix small file output with bz2 compression
e20f169

3
fd915f4

4
120e410

3
8f7b08d

morningman force-pushed the fix_small_bz2 branch from 04d7467 to 8f7b08dCompare September 25, 2025 05:55

CalvinKirs approved these changes Sep 26, 2025
View reviewed changes

github-actionsBot added the approved Indicates a PR has been approved by one committer. label Sep 26, 2025

github-actionsBot added the reviewed label Sep 26, 2025

zy-kkk approved these changes Sep 26, 2025
View reviewed changes

morningman merged commit c861de2 into apache:masterSep 26, 2025
26 of 28 checks passed

github-actionsBot added dev/3.0.x-conflict dev/2.1.x-conflict dev/3.1.x-conflict labels Sep 26, 2025

github-actionsBot mentioned this pull request Sep 26, 2025
branch-4.0: [fix](outfile) fix small file output with bz2 compression #56368 #56495
Merged

CalvinKirs pushed a commit that referenced this pull request Sep 26, 2025
branch-4.0: [fix](outfile) fix small file output with bz2 compression#…
f2d5f2a
…56368 (#56495) Cherry-picked from #56368 Co-authored-by: Mingyu Chen (Rayner) <morningman@163.com>

morningman added dev/4.0.0-merged and removed dev/4.0.x labels Sep 28, 2025

morningman mentioned this pull request Oct 16, 2025
branch-3.1: [fix](outfile) fix small file output with bz2 compression #56368 #57041
Merged

morningman added a commit that referenced this pull request Oct 16, 2025
branch-3.1: [fix](outfile) fix small file output with bz2 compression#…
cb07d50
…56368 (#57041) bp #56368

morningman added dev/3.1.2-merged and removed dev/3.1.x dev/3.1.x-conflict labels Oct 16, 2025

Hastyshell pushed a commit to Hastyshell/doris that referenced this pull request Nov 12, 2025
[fix](outfile) fix small file output with bz2 compression (apache#56368…
0e382c4
…) (apache#4824) closeapache#4791

Uh oh!

Conversation

morningman commented Sep 23, 2025• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What problem does this PR solve?

Release note

Check List (For Author)

Check List (For Reviewer who merge this PR)

Uh oh!

hello-stephen commented Sep 23, 2025

Uh oh!

morningman commented Sep 23, 2025

Uh oh!

doris-robot commented Sep 23, 2025

Uh oh!

hello-stephen commented Sep 23, 2025

BE UT Coverage Report

Uh oh!

hello-stephen commented Sep 23, 2025

BE Regression && UT Coverage Report

Uh oh!

morningman commented Sep 24, 2025

Uh oh!

doris-robot commented Sep 24, 2025

Uh oh!

doris-robot commented Sep 24, 2025

BE UT Coverage Report

Uh oh!

morningman commented Sep 25, 2025

Uh oh!

doris-robot commented Sep 25, 2025

Uh oh!

hello-stephen commented Sep 25, 2025

BE UT Coverage Report

Uh oh!

morningman commented Sep 25, 2025

Uh oh!

doris-robot commented Sep 25, 2025

Uh oh!

doris-robot commented Sep 25, 2025

BE UT Coverage Report

Uh oh!

hello-stephen commented Sep 25, 2025

BE Regression && UT Coverage Report

Uh oh!

github-actionsBot commented Sep 26, 2025

Uh oh!

github-actionsBot commented Sep 26, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

morningman commented Sep 23, 2025•
edited
Loading