Skip to content

[feature](be) Add adaptive batch size for scan path (#62835) - #63005

Merged
yiguolei merged 2 commits into
apache:branch-4.1from
mrhhsg:pick_abs
May 11, 2026
Merged

[feature](be) Add adaptive batch size for scan path (#62835)#63005
yiguolei merged 2 commits into
apache:branch-4.1from
mrhhsg:pick_abs

Conversation

@mrhhsg

@mrhhsgmrhhsg commented May 6, 2026

Copy link
Copy Markdown
Member

Pick PR: #62835

Problem Summary: Add adaptive block row prediction for SegmentIterator, OLAP scan, file scan, and format readers. The scan path now uses a row ceiling plus preferred output byte budget to reduce oversized blocks for wide rows while preserving row-limited behavior for narrow rows. This commit also introduces the shared session/config/thrift/runtime budget plumbing used by later operators.

Adds adaptive batch size controls for scan output blocks: preferred_block_size_bytes and preferred_max_column_in_block_size_bytes.

  • Test: Unit Test
  • Unit Test: ./run-be-ut.sh --run --filter=BlockBudgetTest.:RuntimeStateBatchSizeTest.:RuntimeStateBlockSizeBytesTest.:RuntimeStateMaxColBytesTest.:MockRuntimeStateBlockBudgetTest.:AdaptiveBlockSizePredictorTest.:BlockReaderBatchMaxRowsTest.:EstimateCollectedEnoughTest.:CollectedEnoughWithColumnsTest.:BlockReaderByteBudgetTest.:SegmentColumnRawDataBytesTest.:CsvReaderSetBatchSizeTest.:NewJsonReaderSetBatchSizeTest.:OrcReaderTest.:TableFormatReaderTest.:ProfileSpecTest.:LocalExchangerTest.*
  • Behavior changed: Yes (scan output block sizing can now be byte-budget limited when adaptive batch size is enabled)
  • Does this need documentation: Yes

Issue Number: close #xxx

Related PR: #xxx

Problem Summary:

None

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:

  • This is a refactor/code format and no logic has been changed.
    - [ ] Previous test can cover this change. - [ ] No code files have been changed. - [ ] Other reason

  • Behavior changed:

    • No.
    • Yes.

  • Does this need documentation?

    • No.

  • Yes.

  • Confirm the release note

  • Confirm test cases

  • Confirm document

  • Add branch pick label

What problem does this PR solve?

Issue Number: close #xxx

Related PR: #xxx

Problem Summary:

Release note

None

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason

  • Behavior changed:

    • No.
    • Yes.

  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@mrhhsg
mrhhsg requested a review from yiguolei as a code ownerMay 6, 2026 06:43
@hello-stephen

hello-stephen commented May 6, 2026

Copy link
Copy Markdown
Contributor

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@mrhhsg

mrhhsg commented May 6, 2026

Copy link
Copy Markdown
MemberAuthor

run buildall

@mrhhsg

mrhhsg commented May 9, 2026

Copy link
Copy Markdown
MemberAuthor

run buildall

@hello-stephen

hello-stephen commented May 9, 2026

Copy link
Copy Markdown
Contributor

FE UT Coverage Report

Increment line coverage 83.33% (5/6) 🎉
Increment coverage report
Complete coverage report

@hello-stephen

hello-stephen commented May 9, 2026

Copy link
Copy Markdown
Contributor

BE Regression && UT Coverage Report

Increment line coverage 86.06% (358/416) 🎉

Increment coverage report
Complete coverage report

CategoryCoverage
Function Coverage73.33% (27069/36913)
Line Coverage56.86% (291632/512884)
Region Coverage54.16% (242731/448210)
Branch Coverage55.89% (105564/188893)

@hello-stephen

hello-stephen commented May 9, 2026

Copy link
Copy Markdown
Contributor

FE Regression Coverage Report

Increment line coverage 66.67% (4/6) 🎉
Increment coverage report
Complete coverage report

@hello-stephen

hello-stephen commented May 9, 2026

Copy link
Copy Markdown
Contributor

BE Regression && UT Coverage Report

Increment line coverage 85.58% (356/416) 🎉

Increment coverage report
Complete coverage report

CategoryCoverage
Function Coverage71.51% (26402/36922)
Line Coverage54.44% (279470/513332)
Region Coverage51.53% (231282/448864)
Branch Coverage53.03% (100273/189092)

@hello-stephen

hello-stephen commented May 9, 2026

Copy link
Copy Markdown
Contributor

BE Regression && UT Coverage Report

Increment line coverage 85.58% (356/416) 🎉

Increment coverage report
Complete coverage report

CategoryCoverage
Function Coverage71.50% (26398/36922)
Line Coverage54.44% (279462/513332)
Region Coverage51.54% (231338/448864)
Branch Coverage53.03% (100267/189092)

@hello-stephen

hello-stephen commented May 9, 2026

Copy link
Copy Markdown
Contributor

FE Regression Coverage Report

Increment line coverage 66.67% (4/6) 🎉
Increment coverage report
Complete coverage report

@hello-stephen

hello-stephen commented May 9, 2026

Copy link
Copy Markdown
Contributor

BE Regression && UT Coverage Report

Increment line coverage 85.58% (356/416) 🎉

Increment coverage report
Complete coverage report

CategoryCoverage
Function Coverage71.50% (26398/36922)
Line Coverage54.44% (279462/513332)
Region Coverage51.54% (231338/448864)
Branch Coverage53.03% (100267/189092)

@hello-stephen

hello-stephen commented May 9, 2026

Copy link
Copy Markdown
Contributor

FE Regression Coverage Report

Increment line coverage 66.67% (4/6) 🎉
Increment coverage report
Complete coverage report

1 similar comment
@hello-stephen

hello-stephen commented May 9, 2026

Copy link
Copy Markdown
Contributor

FE Regression Coverage Report

Increment line coverage 66.67% (4/6) 🎉
Increment coverage report
Complete coverage report

@mrhhsg
[feature](be) Add adaptive batch size for scan path (apache#62835)
ac28675
Issue Number: None
Related PR: None
Problem Summary: Add adaptive block row prediction for SegmentIterator,
OLAP scan, file scan, and format readers. The scan path now uses a row
ceiling plus preferred output byte budget to reduce oversized blocks for
wide rows while preserving row-limited behavior for narrow rows. This
commit also introduces the shared session/config/thrift/runtime budget
plumbing used by later operators.
Adds adaptive batch size controls for scan output blocks:
preferred_block_size_bytes and preferred_max_column_in_block_size_bytes.
- Test: Unit Test
- Unit Test: ./run-be-ut.sh --run
--filter=BlockBudgetTest.*:RuntimeStateBatchSizeTest.*:RuntimeStateBlockSizeBytesTest.*:RuntimeStateMaxColBytesTest.*:MockRuntimeStateBlockBudgetTest.*:AdaptiveBlockSizePredictorTest.*:BlockReaderBatchMaxRowsTest.*:EstimateCollectedEnoughTest.*:CollectedEnoughWithColumnsTest.*:BlockReaderByteBudgetTest.*:SegmentColumnRawDataBytesTest.*:CsvReaderSetBatchSizeTest.*:NewJsonReaderSetBatchSizeTest.*:OrcReaderTest.*:TableFormatReaderTest.*:ProfileSpecTest.*:LocalExchangerTest.*
- Behavior changed: Yes (scan output block sizing can now be byte-budget
limited when adaptive batch size is enabled)
- Does this need documentation: Yes
Issue Number: close #xxx
Related PR: #xxx
Problem Summary:
None
- Test <!-- At least one of them must be included. -->
- [ ] Regression test
- [ ] Unit Test
- [ ] Manual test (add detailed scripts or steps below)
- [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
- [ ] Previous test can cover this change.
- [ ] No code files have been changed.
- [ ] Other reason <!-- Add your reason? -->
- Behavior changed:
- [ ] No.
- [ ] Yes. <!-- Explain the behavior change -->
- Does this need documentation?
- [ ] No.
- [ ] Yes. <!-- Add document PR link here. eg:
apache/doris-website#1214 -->
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label <!-- Add branch pick label that this PR
should merge into -->
---------
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@mrhhsg
mrhhsgforce-pushed the pick_abs branch 2 times, most recently from 7b81191 to af17679CompareMay 10, 2026 14:02
@mrhhsg
[fix](be) Deduplicate cluster key MOW compaction output
16fd978
Issue Number: None
Related PR: None
Problem Summary: Cluster-key MOW compaction sorts rows by cluster key, so duplicate unique keys may be non-adjacent and can remain visible in the output rowset. Scan the output rowset primary key index after compaction and add output-rowset internal delete bitmap entries for older duplicate unique-key rows.
None
- Test: Unit Test
- Ran ./run-be-ut.sh --run --filter=VerticalCompactionTest.ClusterKeyMowCompactionNeedsOutputRowsetInternalDedup -j 8
- Behavior changed: No
- Does this need documentation: No
@hello-stephen

hello-stephen commented May 10, 2026

Copy link
Copy Markdown
Contributor

FE Regression Coverage Report

Increment line coverage 66.67% (4/6) 🎉
Increment coverage report
Complete coverage report

1 similar comment
@hello-stephen

hello-stephen commented May 10, 2026

Copy link
Copy Markdown
Contributor

FE Regression Coverage Report

Increment line coverage 66.67% (4/6) 🎉
Increment coverage report
Complete coverage report

@hello-stephen

hello-stephen commented May 11, 2026

Copy link
Copy Markdown
Contributor

BE Regression && UT Coverage Report

Increment line coverage 83.74% (546/652) 🎉

Increment coverage report
Complete coverage report

CategoryCoverage
Function Coverage71.46% (26384/36919)
Line Coverage54.54% (280001/513416)
Region Coverage51.80% (232630/449067)
Branch Coverage53.15% (100563/189200)

@yiguoleiyiguolei reopened this May 11, 2026
@github-actionsgithub-actionsBot added the approved Indicates a PR has been approved by one committer. label May 11, 2026
@github-actions

github-actionsBot commented May 11, 2026

Copy link
Copy Markdown
Contributor

PR approved by at least one committer and no changes requested.

@github-actions

github-actionsBot commented May 11, 2026

Copy link
Copy Markdown
Contributor

PR approved by anyone and no changes requested.

@hello-stephen

hello-stephen commented May 11, 2026

Copy link
Copy Markdown
Contributor

skip buildall

@yiguolei
yiguolei merged commit 83d9e70 into apache:branch-4.1May 11, 2026
46 of 50 checks passed
@mrhhsgmrhhsg mentioned this pull request May 15, 2026
Merged
yiguolei pushed a commit that referenced this pull request May 15, 2026
@mrhhsg
[fix](scan) Fix adaptive load batch sizing (#63245)
f27a1e5
## Summary
Related PR: #63005
- update the file scan load path to feed converted blocks into adaptive
batch size prediction before and after varchar truncation
## Root Cause
The load path initialized adaptive batch sizing but did not update the
predictor after converting loaded source blocks. As a result, the reader
could keep using the initial small probe size.
## Validation
- `git diff --check`
- `git diff --cached --check`
- `./run-regression-test.sh --run -d unique_with_mow_c_p0 -s
test_compact_with_seq2`
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@yiguoleiyiguoleiyiguolei approved these changes

Assignees

No one assigned

Labels

approvedIndicates a PR has been approved by one committer.reviewed

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@mrhhsg@hello-stephen@yiguolei