Skip to content

branch-4.1: [feature](be) Support zstd stream load compression #64711 - #64750

Merged
yiguolei merged 1 commit into
branch-4.1from
auto-pick-64711-branch-4.1
Jun 24, 2026
Merged

branch-4.1: [feature](be) Support zstd stream load compression #64711#64750
yiguolei merged 1 commit into
branch-4.1from
auto-pick-64711-branch-4.1

Conversation

@github-actions

@github-actionsgithub-actionsBot commented Jun 23, 2026

Copy link
Copy Markdown
Contributor

Cherry-picked from #64711

@0AyanamiReiYour Name
[feature](be) Support zstd stream load compression (#64711)
f8161c6
Stream load does not support `compress_type=zstd` in the shared load
format parser. Async group commit also checks only legacy compressed CSV
format enum values when estimating compressed input size, so
`compress_type` based compressed input is not handled consistently by
stream load and HTTP stream load.
This PR adds ZSTD parsing in `LoadUtil::parse_format`, adds a shared
`LoadUtil::is_compressed_load` helper for `compress_type` and legacy
compressed CSV format types, and uses it in stream load and HTTP stream
group commit paths. This PR also adds BE UT and regression coverage for
ZSTD CSV/JSON stream load and group commit stream/HTTP stream load.
@github-actions
github-actionsBot requested a review from yiguolei as a code ownerJune 23, 2026 13:24
@hello-stephen

hello-stephen commented Jun 23, 2026

Copy link
Copy Markdown
Contributor

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@hello-stephen

hello-stephen commented Jun 23, 2026

Copy link
Copy Markdown
Contributor

run buildall

@hello-stephen

hello-stephen commented Jun 23, 2026

Copy link
Copy Markdown
Contributor

BE Regression && UT Coverage Report

Increment line coverage 92.31% (36/39) 🎉

Increment coverage report
Complete coverage report

CategoryCoverage
Function Coverage73.66% (27421/37225)
Line Coverage57.35% (297200/518225)
Region Coverage55.15% (250229/453691)
Branch Coverage56.41% (108060/191565)

@yiguolei

yiguolei commented Jun 24, 2026

Copy link
Copy Markdown
Contributor

skip buildall

@yiguolei
yiguolei merged commit a9fb05d into branch-4.1Jun 24, 2026
38 of 41 checks passed
@morningman
morningman deleted the auto-pick-64711-branch-4.1 branch July 8, 2026 14:10
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@yiguoleiyiguoleiyiguolei approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@hello-stephen@yiguolei@0AyanamiRei