Uh oh!
There was an error while loading. Please reload this page.
[feature](be) Add adaptive batch size for scan path - #62835
Merged
Conversation
Thearas
commented
Apr 25, 2026
Contributor
Thank you for your contribution to Apache Doris. Please clearly describe your PR:
|
mrhhsg
commented
Apr 25, 2026
MemberAuthor
/review |
Contributor
OpenCode automated review failed and did not complete. Error: Review step was failure (possibly timeout or cancelled) Please inspect the workflow logs and rerun the review after the underlying issue is resolved. |
Contributor
There was a problem hiding this comment.
Pull request overview
Adds adaptive (byte-budgeted) batch sizing to the scan path so output blocks can be limited by a preferred byte size (and per-column byte cap) while still respecting the existing row ceiling.
Changes:
- Plumbs new session/query options
preferred_block_size_bytesandpreferred_max_column_in_block_size_bytesfrom FE session variables through Thrift into BE runtime/scanners. - Implements adaptive block sizing in storage (SegmentIterator predictor + raw-bytes metadata) and in external file scans (FileScanner-driven batch size adjustment).
- Adds extensive BE/FE unit tests plus a new regression suite for correctness with adaptive batching enabled/disabled.
Reviewed changes
Copilot reviewed 70 out of 71 changed files in this pull request and generated 9 comments.
Show a summary per file
| File | Description |
|---|---|
| regression-test/suites/unique_with_mow_c_p0/test_schema_change_add_key_column.groovy | Disables adaptive batch size during suite to preserve prior row-byte behavior. |
| regression-test/suites/unique_with_mow_c_p0/test_compact_multi_segments.groovy | Disables adaptive batch size during suite to preserve prior row-byte behavior. |
| regression-test/suites/query_p0/adaptive_batch_size/adaptive_batch_size.groovy | New regression coverage for correctness with adaptive batching toggled. |
| regression-test/suites/fault_injection_p0/test_skip_calc_between_segments.groovy | Temporarily disables adaptive batch size via BE config in fault injection. |
| regression-test/data/query_p0/adaptive_batch_size/adaptive_batch_size.out | Golden output for the new adaptive batch size regression suite. |
| gensrc/thrift/PaloInternalService.thrift | Adds Thrift query options for preferred block size bytes and per-column cap. |
| fe/fe-core/src/test/java/org/apache/doris/qe/VariableMgrTest.java | FE tests for defaults/forwarding/validation of new session variables. |
| fe/fe-core/src/main/java/org/apache/doris/qe/SessionVariable.java | Defines new session vars and forwards them to Thrift query options. |
| fe/be-java-extensions/java-common/src/test/java/org/apache/doris/common/jni/JniScannerTest.java | Tests Java scanner batch-size mutability across reads. |
| fe/be-java-extensions/java-common/src/main/java/org/apache/doris/common/jni/JniScanner.java | Adds setBatchSize() to support adaptive row batching from BE. |
| fe/be-java-extensions/hadoop-hudi-scanner/src/main/java/org/apache/doris/hudi/HadoopHudiJniScanner.java | Uses dynamic batchSize in getNext() loop. |
| be/test/util/profile_spec_test.cpp | Tests output block byte counters update on operator output blocks. |
| be/test/testutil/mock/mock_runtime_state.h | Overrides preferred byte budgets to bypass clamping in BE tests. |
| be/test/storage/test_data/tablet_meta_test.hdr | Adds test data artifact for tablet meta. |
| be/test/storage/segment/segment_column_raw_data_bytes_test.cpp | Tests segment raw_data_bytes lookup behavior. |
| be/test/storage/segment/mock/mock_segment.h | Adds setter for raw-bytes metadata in segment mock. |
| be/test/storage/segment/adaptive_block_size_predictor_test.cpp | Unit tests for adaptive predictor EWMA + constraints + hinting. |
| be/test/storage/iterator/vcollect_iterator_collected_enough_test.cpp | Tests byte-budget “collected enough” logic for VCollectIterator/BlockReader. |
| be/test/storage/iterator/block_reader_batch_max_rows_test.cpp | Tests batch_max_rows vs byte budget behavior in BlockReader. |
| be/test/runtime/runtime_state_block_budget_test.cpp | Tests RuntimeState batch size and preferred byte budget clamping/behavior. |
| be/test/format/table/table_format_reader_test.cpp | Tests synthesized-column handler plumbing. |
| be/test/format/orc/orc_reader_test.cpp | Tests ORC reader batch rebuild on batch size change. |
| be/test/format/json/json_reader_test.cpp | Tests JSON reader batch size setter/getter via GenericReader. |
| be/test/format/csv/csv_reader_test.cpp | Tests CSV reader set_batch_size calls don’t crash. |
| be/test/exec/pipeline/local_exchanger_test.cpp | Adjusts LocalExchangeSourceLocalState construction to use runtime state. |
| be/test/common/block_budget_test.cpp | Unit tests for new BlockBudget helper. |
| be/src/util/block_budget.h | Introduces BlockBudget utility for combined row+byte limiting. |
| be/src/storage/tablet/tablet_reader.h | Adds preferred byte budget virtual API + setters in TabletReader. |
| be/src/storage/tablet/tablet_reader.cpp | Preserves original requested output columns for later projection mapping. |
| be/src/storage/segment/segment_iterator.h | Adds AdaptiveBlockSizePredictor to SegmentIterator state. |
| be/src/storage/segment/segment_iterator.cpp | Predicts per-batch row count from byte budgets and updates EWMA post-batch. |
| be/src/storage/segment/segment.h | Adds raw-bytes metadata cache accessors and const tablet_schema(). |
| be/src/storage/segment/segment.cpp | Caches per-column raw_data_bytes during footer/meta creation. |
| be/src/storage/segment/adaptive_block_size_predictor.h | Declares predictor and metadata hint computation APIs. |
| be/src/storage/segment/adaptive_block_size_predictor.cpp | Implements EWMA update and next-row prediction with constraints. |
| be/src/storage/rowset/rowset_reader_context.h | Stores preferred byte budgets + origin_return_columns in reader context. |
| be/src/storage/rowset/beta_rowset_reader.cpp | Passes byte budget + output column ids into StorageReadOptions. |
| be/src/storage/olap_common.h | Adds stats fields to record predicted min/max rows across batches. |
| be/src/storage/iterators.h | Extends StorageReadOptions with adaptive byte budgets and output column list. |
| be/src/storage/iterator/vcollect_iterator.h | Exposes estimate_collected_enough helper + topN chunking members. |
| be/src/storage/iterator/vcollect_iterator.cpp | Adds byte-budget-aware early stop + topN chunking under byte budget. |
| be/src/storage/iterator/block_reader.h | Overrides preferred_block_size_bytes() + adds byte-budget helper. |
| be/src/storage/iterator/block_reader.cpp | Adds periodic byte-budget checks in key paths and uses batch_max_rows(). |
| be/src/runtime/runtime_state.h | Adds clamped accessors for preferred byte budgets and clamps batch_size(). |
| be/src/runtime/runtime_profile_counter_names.h | Adds common counters for output block bytes and min/max. |
| be/src/format/parquet/vparquet_reader.h | Exposes set_batch_size()/get_batch_size() overrides for adaptive control. |
| be/src/format/parquet/vparquet_reader.cpp | Implements ParquetReader::set_batch_size. |
| be/src/format/orc/vorc_reader.h | Adds set_batch_size()/get_batch_size() override declarations. |
| be/src/format/orc/vorc_reader.cpp | Rebuilds ORC row batch when batch size changes. |
| be/src/format/json/new_json_reader.h | Adds set_batch_size()/get_batch_size() overrides for FileScanner control. |
| be/src/format/json/new_json_reader.cpp | Uses preferred_block_size_bytes guard and configurable _batch_size. |
| be/src/format/jni/jni_reader.h | Adds set_batch_size()/get_batch_size() for JNI readers. |
| be/src/format/jni/jni_reader.cpp | Calls Java setBatchSize() at runtime when scanner is open. |
| be/src/format/generic_reader.h | Adds virtual set_batch_size/get_batch_size extension points. |
| be/src/format/csv/csv_reader.h | Adds set_batch_size and stores adaptive batch size. |
| be/src/format/csv/csv_reader.cpp | Uses preferred_block_size_bytes sampling guard + configurable batch size. |
| be/src/exec/scan/scanner.cpp | Uses preferred_block_size_bytes when merging padding blocks. |
| be/src/exec/scan/olap_scanner.cpp | Passes byte budgets to TabletReader and reports predicted min/max rows. |
| be/src/exec/scan/file_scanner.h | Adds adaptive predictor state and new profile counters. |
| be/src/exec/scan/file_scanner.cpp | Implements adaptive row prediction for file readers and related counters. |
| be/src/exec/operator/schema_scan_operator.h | Moves local state ctor out-of-line; updates block stop condition wiring. |
| be/src/exec/operator/schema_scan_operator.cpp | Uses BlockBudget exceeded() for schema scan block accumulation. |
| be/src/exec/operator/scan_operator.h | Uses shared output-block counter updater. |
| be/src/exec/operator/operator.h | Adds BlockBudget to local state and common output-block byte counters. |
| be/src/exec/operator/operator.cpp | Initializes BlockBudget + output byte counters and updates after projects. |
| be/src/exec/operator/olap_scan_operator.h | Adds profile counters for adaptive predicted min/max rows. |
| be/src/exec/operator/olap_scan_operator.cpp | Initializes adaptive predicted min/max profile counters. |
| be/src/exec/operator/mock_scan_operator.h | Adds ability to emit a controlled output block for profile tests. |
| be/src/core/block/block.h | Clarifies bytes() semantics and adds columns_byte_size helper for mutables. |
| be/src/common/config.h | Adds BE config flag enable_adaptive_batch_size. |
| be/src/common/config.cpp | Defaults enable_adaptive_batch_size to true. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
+238
to
+243
| bool FileScanner::_should_enable_adaptive_batch_size(TFileFormatType::type format_type) const { | ||
| // Only enable for readers that support set_batch_size(). | ||
| // Table-format wrappers are covered because they delegate to native readers. | ||
| if (_state->preferred_block_size_bytes() == 0) { | ||
| return false; | ||
| } |
There was a problem hiding this comment.
_should_enable_adaptive_batch_size only checks preferred_block_size_bytes() == 0, but RuntimeState currently returns a non-zero value even when config::enable_adaptive_batch_size is false. That means adaptive batching may still run when the BE config is disabled. Please gate this on config::enable_adaptive_batch_size (or rely on preferred_block_size_bytes() returning 0 when disabled) so the config flag reliably disables the feature.
| // Predict how many more rows can be appended before hitting max_block_bytes. | ||
| const auto remaining_bytes = max_block_bytes - bytes; | ||
| const auto bytes_per_row = bytes / rows; |
There was a problem hiding this comment.
bytes_per_row is computed as bytes / rows and can evaluate to 0 when bytes>0 but bytes<rows (e.g. constant/low-footprint columns), which would make remaining_bytes / bytes_per_row divide by zero. Please clamp bytes_per_row to at least 1 (or handle bytes_per_row==0) before computing remaining_rows.
Suggested change
| constauto bytes_per_row = bytes / rows; | |
| constauto bytes_per_row = std::max<size_t>(1, bytes / rows); |
Comment on lines
50
to
54
| onFinish { | ||
| GetDebugPoint().disableDebugPointForAllBEs("MemTable.need_flush") | ||
| set_original_be_param("doris_scanner_row_bytes", backendId_to_params) | ||
| set_be_param.call("enable_adaptive_batch_size", "true") | ||
| } |
There was a problem hiding this comment.
onFinish unconditionally sets enable_adaptive_batch_size=true, but the suite disables it during the test. This can leak state into subsequent tests because it doesn’t restore the prior value. Please save the original enable_adaptive_batch_size BE param and restore it in onFinish (or use reset_be_param).
Comment on lines
+150
to
+185
| // Target byte budget per output block (default 8MB when adaptive is enabled). | ||
| // The public FE/session contract is [1MB, 512MB]; this accessor still clamps any direct | ||
| // thrift or mixed-version out-of-range value into that range. Returns `kMax` only when adaptive | ||
| // is disabled by BE config. | ||
| MOCK_FUNCTION size_t preferred_block_size_bytes() const { | ||
| static constexpr int64_t kDefault = 8388608L; // 8MB | ||
| static constexpr int64_t kMax = 536870912L; // 512MB | ||
| static constexpr int64_t kMin = 1048576L; // 1MB | ||
| if (!config::enable_adaptive_batch_size) [[unlikely]] { | ||
| return kMax; | ||
| } | ||
| if (_query_options.__isset.preferred_block_size_bytes) [[likely]] { | ||
| return std::max<int64_t>( | ||
| kMin, std::min<int64_t>(_query_options.preferred_block_size_bytes, kMax)); | ||
| } | ||
| return kDefault; | ||
| } | ||
| // Per-column byte limit for adaptive chunk rows (default 1MB when adaptive is enabled). | ||
| // The public FE/session contract is [256KB, 128MB]; this accessor still clamps any direct | ||
| // thrift or mixed-version out-of-range value into that range. Returns `kMax` only when adaptive | ||
| // is disabled by BE config. | ||
| MOCK_FUNCTION size_t preferred_max_column_in_block_size_bytes() const { | ||
| static constexpr int64_t kDefault = 1048576L; // 1MB | ||
| static constexpr int64_t kMax = 134217728L; // 128MB | ||
| static constexpr int64_t kMin = 262144L; // 256KB | ||
| if (!config::enable_adaptive_batch_size) [[unlikely]] { | ||
| return kMax; | ||
| } | ||
| if (_query_options.__isset.preferred_max_column_in_block_size_bytes) [[likely]] { | ||
| return std::max<int64_t>( | ||
| kMin, std::min<int64_t>(_query_options.preferred_max_column_in_block_size_bytes, | ||
| kMax)); | ||
| } | ||
| return kDefault; | ||
| } |
There was a problem hiding this comment.
preferred_block_size_bytes()/preferred_max_column_in_block_size_bytes() return the max clamp when config::enable_adaptive_batch_size is false. Several call sites (e.g. FileScanner::_should_enable_adaptive_batch_size) use a return value of 0 to mean “disabled”, so returning 512MB/128MB can inadvertently keep adaptive logic active and change behavior even when the BE config is off. Recommend returning 0 when adaptive is disabled (and updating call sites/tests accordingly), keeping clamping only for non-zero values.
Comment on lines
+247
to
262
| void NewJsonReader::set_batch_size(size_t batch_size) { | ||
| _batch_size = batch_size; | ||
| } | ||
| Status NewJsonReader::_do_get_next_block(Block* block, size_t* read_rows, bool* eof) { | ||
| if (_reader_eof) { | ||
| *eof = true; | ||
| return Status::OK(); | ||
| } | ||
| const int batch_size = std::max(_state->batch_size(), (int)_MIN_BATCH_SIZE); | ||
| const int64_t max_block_bytes = | ||
| (_state->query_type() == TQueryType::LOAD && config::load_reader_max_block_bytes > 0) | ||
| ? config::load_reader_max_block_bytes | ||
| : 0; | ||
| const auto batch_size = _batch_size; | ||
| const auto max_block_bytes = _state->preferred_block_size_bytes(); | ||
| while (block->rows() < batch_size && !_reader_eof && | ||
| (max_block_bytes <= 0 || (int64_t)block->bytes() < max_block_bytes)) { | ||
| size_t next_checking_rows = *read_rows + 1; | ||
| while (block->rows() < batch_size && !_reader_eof) { | ||
| if (UNLIKELY(_read_json_by_line && _skip_first_line)) { |
There was a problem hiding this comment.
set_batch_size() allows setting _batch_size to 0, but _do_get_next_block uses while (block->rows() < batch_size ...). If batch_size becomes 0, this loop never runs and the method returns without setting eof, which can cause the scanner to spin with empty blocks. Please enforce batch_size >= 1 (or treat 0 as “reset to default”, e.g. max(state->batch_size(), _MIN_BATCH_SIZE)).
| } | ||
| const auto remaining_bytes = max_block_bytes - bytes; | ||
| const auto bytes_per_row = bytes / block->rows(); |
There was a problem hiding this comment.
bytes_per_row is computed as bytes / block->rows() and can become 0 for low-byte constant columns (bytes>0 but bytes<rows), which would lead to division-by-zero when computing remaining_rows. Please clamp bytes_per_row to at least 1 (similar to other places using avg_row_bytes) before dividing, or handle the 0 case explicitly.
Suggested change
| constauto bytes_per_row = bytes / block->rows(); | |
| constauto bytes_per_row = std::max<size_t>(1, bytes / block->rows()); |
Comment on lines
+383
to
+395
| void CsvReader::set_batch_size(size_t batch_size) { | ||
| _batch_size = batch_size; | ||
| } | ||
| // !FIXME: Here we should use MutableBlock | ||
| Status CsvReader::_do_get_next_block(Block* block, size_t* read_rows, bool* eof) { | ||
| if (_line_reader_eof) { | ||
| *eof = true; | ||
| return Status::OK(); | ||
| } | ||
| const int batch_size = std::max(_state->batch_size(), (int)_MIN_BATCH_SIZE); | ||
| const int64_t max_block_bytes = | ||
| (_state->query_type() == TQueryType::LOAD && config::load_reader_max_block_bytes > 0) | ||
| ? config::load_reader_max_block_bytes | ||
| : 0; | ||
| const size_t batch_size = _batch_size; | ||
| const auto max_block_bytes = _state->preferred_block_size_bytes(); |
There was a problem hiding this comment.
set_batch_size() stores the value directly into _batch_size, and _do_get_next_block uses it as the loop upper bound. If _batch_size is set to 0 (the header comment says 0 means “not set”), the reader will return empty blocks and also sets *eof = (rows == 0) which can incorrectly signal EOF. Please enforce batch_size >= 1 (or interpret 0 as “use state->batch_size()/default”) inside set_batch_size() or before the read loop.
Comment on lines
33
to
36
| #include "agent/be_exec_version_manager.h" | ||
| #include "common/config.h" | ||
| #include "common/status.h" | ||
| #include "exprs/function_filter.h" |
There was a problem hiding this comment.
common/config.h is included here but not used anywhere in this header (only referenced in a comment). Consider dropping the include to reduce transitive dependencies and compile time.
Comment on lines
60
to
64
| onFinish { | ||
| GetDebugPoint().clearDebugPointsForAllBEs() | ||
| set_original_be_param("doris_scanner_row_bytes", backendId_to_params) | ||
| set_be_param.call("enable_adaptive_batch_size", "true") | ||
| } |
There was a problem hiding this comment.
onFinish unconditionally sets enable_adaptive_batch_size=true, but the test body sets it to false. This doesn’t restore the original BE config and can leak into later suites (especially if a prior suite intentionally disabled it). Please capture the original enable_adaptive_batch_size value (similar to doris_scanner_row_bytes) and restore it in onFinish (or call reset_be_param).
mrhhsg added a commit
to mrhhsg/doris
that referenced
this pull request
Apr 25, 2026
### What problem does this PR solve? Issue Number: None Related PR: apache#62835 Problem Summary: Address review comments on the adaptive batch size scan-path PR: - runtime_state: preferred_block_size_bytes() and preferred_max_column_in_block_size_bytes() now return 0 when config::enable_adaptive_batch_size is false. This matches the contract used by callers (BlockBudget, BlockReader, FileScanner) which treat 0 as 'adaptive disabled / no byte budget' and ensures the BE config flag reliably disables the feature end-to-end. - scanner.cpp: the padding loop in get_block_after_projects now treats block_max_bytes == 0 as 'no byte ceiling' so disabling adaptive batch size does not skip block padding entirely. - csv_reader / new_json_reader: clamp set_batch_size() to at least _MIN_BATCH_SIZE so a stray 0 cannot freeze the read loop or spuriously signal EOF. - csv_reader / new_json_reader: clamp bytes_per_row to at least 1 in the adaptive predict path so low-footprint constant columns cannot trigger a divide-by-zero. - tablet_reader.h: drop the unused #include "common/config.h". - regression tests test_compact_multi_segments and test_schema_change_add_key_column: capture and restore the original enable_adaptive_batch_size BE param via set_original_be_param so disabling it inside the test does not leak into subsequent suites. ### Release note None ### Check List (For Author) - Test: - Unit Test (BlockBudget / RuntimeState* / CsvReaderSetBatchSize / NewJsonReaderSetBatchSize / BlockReaderBatchMaxRows / AdaptiveBlockSizePredictor / EstimateCollectedEnough / CollectedEnoughWithColumns / BlockReaderByteBudget / SegmentColumnRawDataBytes / OrcReader / TableFormatReader / ProfileSpec / LocalExchanger) - Behavior changed: No - Does this need documentation: No Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Contributor
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 70 out of 71 changed files in this pull request and generated 3 comments.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
+205
to
+210
| } finally { | ||
| reset_be_param("enable_adaptive_batch_size") | ||
| sql "set preferred_block_size_bytes = 8388608" | ||
| sql "set preferred_max_column_in_block_size_bytes = 1048576" | ||
| sql "set batch_size = 8160" | ||
| } |
There was a problem hiding this comment.
reset_be_param("enable_adaptive_batch_size") is not defined in the regression-test Suite DSL (only get_be_param / set_be_param / set_original_be_param exist), so this suite will fail at runtime with a missing-method/property error. Capture the original BE config via get_be_param("enable_adaptive_batch_size") (like other suites in this PR) and restore it in finally using set_original_be_param, or wrap the body in setBeConfigTemporary([enable_adaptive_batch_size:false]) so cleanup is guaranteed.
Comment on lines
+366
to
+370
| // Default value of preferred_block_size_bytes in reader context is 8MB. | ||
| TEST_F(BlockReaderByteBudgetTest, DefaultIsZero) { | ||
| config::enable_adaptive_batch_size = true; | ||
| BlockReader reader; | ||
| EXPECT_EQ(reader.preferred_block_size_bytes(), 8388608UL); |
There was a problem hiding this comment.
The test name DefaultIsZero is misleading: the assertion expects the default byte budget to be 8MB when adaptive batch size is enabled. Rename the test to reflect the expected behavior (e.g. DefaultIs8MB / DefaultIsDefaultBudget) so failures are easier to interpret.
Comment on lines
+1300
to
1303
| // 8192 minus 16 + 16 bytes padding that in padding pod array. | ||
| // This remains the row cap for output blocks even when adaptive byte budgeting is enabled. | ||
| @VarAttrDef.VarAttr(name = BATCH_SIZE, fuzzy = true, checker = "checkBatchSize", needForward = true) | ||
| public int batchSize = 8160; |
There was a problem hiding this comment.
The comment explaining batchSize = 8160 looks mathematically inconsistent: “8192 minus 16 + 16” equals 8192, not 8160. Please correct the explanation (likely 8192 minus 16 minus 16, or otherwise clarify where the 32-row reduction comes from) so future readers understand why 8160 is the default cap.
mrhhsg added a commit
to mrhhsg/doris
that referenced
this pull request
Apr 25, 2026
Issue Number: None Related PR: apache#62835 Problem Summary: Address review comments on the adaptive batch size scan-path PR: - runtime_state: preferred_block_size_bytes() and preferred_max_column_in_block_size_bytes() now return 0 when config::enable_adaptive_batch_size is false. This matches the contract used by callers (BlockBudget, BlockReader, FileScanner) which treat 0 as 'adaptive disabled / no byte budget' and ensures the BE config flag reliably disables the feature end-to-end. - scanner.cpp: the padding loop in get_block_after_projects now treats block_max_bytes == 0 as 'no byte ceiling' so disabling adaptive batch size does not skip block padding entirely. - csv_reader / new_json_reader: clamp set_batch_size() to at least _MIN_BATCH_SIZE so a stray 0 cannot freeze the read loop or spuriously signal EOF. - csv_reader / new_json_reader: clamp bytes_per_row to at least 1 in the adaptive predict path so low-footprint constant columns cannot trigger a divide-by-zero. - tablet_reader.h: drop the unused #include "common/config.h". - regression tests test_compact_multi_segments and test_schema_change_add_key_column: capture and restore the original enable_adaptive_batch_size BE param via set_original_be_param so disabling it inside the test does not leak into subsequent suites. None - Test: - Unit Test (BlockBudget / RuntimeState* / CsvReaderSetBatchSize / NewJsonReaderSetBatchSize / BlockReaderBatchMaxRows / AdaptiveBlockSizePredictor / EstimateCollectedEnough / CollectedEnoughWithColumns / BlockReaderByteBudget / SegmentColumnRawDataBytes / OrcReader / TableFormatReader / ProfileSpec / LocalExchanger) - Behavior changed: No - Does this need documentation: No Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
| } | ||
| bool FileScanner::_should_run_adaptive_batch_size() const { | ||
| return _block_size_predictor != nullptr && _get_push_down_agg_type() != TPushAggOp::type::COUNT; |
Contributor
There was a problem hiding this comment.
为什么这个adptive 会跟push agg op 有关系？ 可以注释注明
| // never let _batch_size be 0 because _do_get_next_block uses it as the | ||
| // upper bound of a `while (rows < _batch_size)` loop and a 0 would make | ||
| // the reader return empty blocks and incorrectly signal EOF. | ||
| _batch_size = std::max(_MIN_BATCH_SIZE, batch_size); |
Contributor
There was a problem hiding this comment.
我感觉这个min batch size 是错的。 如果我们保留它，感觉没有办法做到控制bytes了。 所以要不我们先删掉吧
| RETURN_IF_ERROR(_validate_line(Slice(ptr, size), &success)); | ||
| ++rows; | ||
| block_bytes += size; |
| bool is_remove_bom = false; | ||
| if (_push_down_agg_type == TPushAggOp::type::COUNT) { | ||
| while (rows < batch_size && !_line_reader_eof && | ||
| (max_block_bytes <= 0 || (int64_t)block_bytes < max_block_bytes)) { |
MemberAuthor
There was a problem hiding this comment.
TPushAggOp::type::COUNT 没有必要控制 block bytes
| size_t next_checking_rows = 1; | ||
| auto get_block_bytes = [&]() { | ||
| if (_is_load) { | ||
| return block->bytes(); |
MemberAuthor
There was a problem hiding this comment.
query 模式下真正被增量填充的是本地 mutate 出来的 columns 集合，而不是整个 block；load 模式下 block 本身就是主要写入对象，用 block->bytes() 更直接。
| if (!_block_size_predictor->has_history()) { | ||
| COUNTER_UPDATE(_adaptive_batch_probe_count_counter, 1); | ||
| } | ||
| _block_size_predictor->update(block, _adaptive_batch_output_column_ids); |
MemberAuthor
There was a problem hiding this comment.
因为在 file scanner 不像内表有列的统计信息记录在 meta 信息中，所以这里_adaptive_batch_output_column_ids 就是读到的 block 的所有列的 0-N 的映射。
内表里面，需要将 block 中的列和 meta 中的列对应起来，这个字段才有实际的意义。
| break; | ||
| } | ||
| const auto block_max_bytes = state->preferred_block_size_bytes(); | ||
| while (_padding_block.rows() < min_batch_size && |
Contributor
There was a problem hiding this comment.
这段代码可以简化，我们可以适当允许超过block max bytes的方式，把代码写的更加简洁一些，现在这个eos 被设置为true了，然后还可以再设置为false，有点绕了
| _io_ctx(io_ctx), | ||
| _io_ctx_holder(std::move(io_ctx_holder)) { | ||
| _io_ctx_holder(std::move(io_ctx_holder)), | ||
| _batch_size(state ? state->batch_size() |
Contributor
There was a problem hiding this comment.
直接忽略 MIN BATCH SIZE 吧，我已经在让sida 整体下掉这个了
| _io_ctx(io_ctx), | ||
| _io_ctx_holder(std::move(io_ctx_holder)) { | ||
| _io_ctx_holder(std::move(io_ctx_holder)), | ||
| _batch_size(_MIN_BATCH_SIZE) { |
| continue; | ||
| } | ||
| RETURN_IF_ERROR(_fill_dest_columns(Slice(ptr, size), block, columns, &rows)); | ||
| block_bytes += size; |
Contributor
There was a problem hiding this comment.
我感觉之前的逻辑，根据读取的size 来判断可能是对的，比我们基于block bytes 来判断更好一些
MemberAuthor
There was a problem hiding this comment.
每读一行数据都会马上解析填充到 block 中，读到的 size 和填充后的 block 的 bytes 不一定一致，有可能填充后的 block 会变大
| if (UNLIKELY(_read_json_by_line && _skip_first_line)) { | ||
| size_t size = 0; | ||
| const uint8_t* line_ptr = nullptr; | ||
| RETURN_IF_ERROR(_line_reader->read_line(&line_ptr, &size, &_reader_eof, _io_ctx)); |
| // is disabled by BE config so the value is always a legal byte budget; callers that need | ||
| // to know whether adaptive batch size is active should test | ||
| // `config::enable_adaptive_batch_size` explicitly. | ||
| MOCK_FUNCTION size_t preferred_max_column_in_block_size_bytes() const { |
MemberAuthor
There was a problem hiding this comment.
单列过大更容易造成 cache miss，可以通过这个值让以整列数据全部落在 L2 缓存里
mrhhsg added a commit
to mrhhsg/doris
that referenced
this pull request
Apr 27, 2026
Issue Number: None Related PR: apache#62835 Problem Summary: Address review comments on the adaptive batch size scan-path PR: - runtime_state: preferred_block_size_bytes() and preferred_max_column_in_block_size_bytes() now return 0 when config::enable_adaptive_batch_size is false. This matches the contract used by callers (BlockBudget, BlockReader, FileScanner) which treat 0 as 'adaptive disabled / no byte budget' and ensures the BE config flag reliably disables the feature end-to-end. - scanner.cpp: the padding loop in get_block_after_projects now treats block_max_bytes == 0 as 'no byte ceiling' so disabling adaptive batch size does not skip block padding entirely. - csv_reader / new_json_reader: clamp set_batch_size() to at least _MIN_BATCH_SIZE so a stray 0 cannot freeze the read loop or spuriously signal EOF. - csv_reader / new_json_reader: clamp bytes_per_row to at least 1 in the adaptive predict path so low-footprint constant columns cannot trigger a divide-by-zero. - tablet_reader.h: drop the unused #include "common/config.h". - regression tests test_compact_multi_segments and test_schema_change_add_key_column: capture and restore the original enable_adaptive_batch_size BE param via set_original_be_param so disabling it inside the test does not leak into subsequent suites. None - Test: - Unit Test (BlockBudget / RuntimeState* / CsvReaderSetBatchSize / NewJsonReaderSetBatchSize / BlockReaderBatchMaxRows / AdaptiveBlockSizePredictor / EstimateCollectedEnough / CollectedEnoughWithColumns / BlockReaderByteBudget / SegmentColumnRawDataBytes / OrcReader / TableFormatReader / ProfileSpec / LocalExchanger) - Behavior changed: No - Does this need documentation: No Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
| // Variant subtree: times selecting doc snapshot all iterator (merge doc snapshot into root) | ||
| RuntimeProfile::Counter* _variant_doc_value_column_iter_count = nullptr; | ||
| RuntimeProfile::HighWaterMarkCounter* _adaptive_batch_predict_min_rows_counter = nullptr; |
Contributor
There was a problem hiding this comment.
为什么这个counter 是一个high water mark？
| } | ||
| // If we already computed the full topN result, return the next chunk. | ||
| if (_topn_result_block.rows() > 0) { |
| // Adaptive batch size: snapshot the initial row limit and create predictor if enabled. | ||
| _initial_block_row_max = _opts.block_row_max; | ||
| if (config::enable_adaptive_batch_size && _opts.preferred_block_size_bytes > 0) { | ||
| // Collect per-column raw byte metadata from the segment footer. |
| } | ||
| _ensure_adaptive_batch_output_column_ids(block); | ||
| // Count a probe only when we actually obtain the first non-empty sample that seeds history. |
Contributor
There was a problem hiding this comment.
predictor ---> update (std::vector)
predicator = new Predicator(vector<size_t>)
delete condition on column 4
1,2,3, 4
hello-stephen
commented
Apr 27, 2026
Contributor
FE Regression Coverage ReportIncrement line coverage |
Issue Number: None Related PR: None Problem Summary: Add adaptive block row prediction for SegmentIterator, OLAP scan, file scan, and format readers. The scan path now uses a row ceiling plus preferred output byte budget to reduce oversized blocks for wide rows while preserving row-limited behavior for narrow rows. This commit also introduces the shared session/config/thrift/runtime budget plumbing used by later operators. Adds adaptive batch size controls for scan output blocks: preferred_block_size_bytes and preferred_max_column_in_block_size_bytes. - Test: Unit Test - Unit Test: ./run-be-ut.sh --run --filter=BlockBudgetTest.*:RuntimeStateBatchSizeTest.*:RuntimeStateBlockSizeBytesTest.*:RuntimeStateMaxColBytesTest.*:MockRuntimeStateBlockBudgetTest.*:AdaptiveBlockSizePredictorTest.*:BlockReaderBatchMaxRowsTest.*:EstimateCollectedEnoughTest.*:CollectedEnoughWithColumnsTest.*:BlockReaderByteBudgetTest.*:SegmentColumnRawDataBytesTest.*:CsvReaderSetBatchSizeTest.*:NewJsonReaderSetBatchSizeTest.*:OrcReaderTest.*:TableFormatReaderTest.*:ProfileSpecTest.*:LocalExchangerTest.* - Behavior changed: Yes (scan output block sizing can now be byte-budget limited when adaptive batch size is enabled) - Does this need documentation: Yes
Issue Number: None Related PR: apache#62835 Problem Summary: Address review comments on the adaptive batch size scan-path PR: - runtime_state: preferred_block_size_bytes() and preferred_max_column_in_block_size_bytes() now return 0 when config::enable_adaptive_batch_size is false. This matches the contract used by callers (BlockBudget, BlockReader, FileScanner) which treat 0 as 'adaptive disabled / no byte budget' and ensures the BE config flag reliably disables the feature end-to-end. - scanner.cpp: the padding loop in get_block_after_projects now treats block_max_bytes == 0 as 'no byte ceiling' so disabling adaptive batch size does not skip block padding entirely. - csv_reader / new_json_reader: clamp set_batch_size() to at least _MIN_BATCH_SIZE so a stray 0 cannot freeze the read loop or spuriously signal EOF. - csv_reader / new_json_reader: clamp bytes_per_row to at least 1 in the adaptive predict path so low-footprint constant columns cannot trigger a divide-by-zero. - tablet_reader.h: drop the unused #include "common/config.h". - regression tests test_compact_multi_segments and test_schema_change_add_key_column: capture and restore the original enable_adaptive_batch_size BE param via set_original_be_param so disabling it inside the test does not leak into subsequent suites. None - Test: - Unit Test (BlockBudget / RuntimeState* / CsvReaderSetBatchSize / NewJsonReaderSetBatchSize / BlockReaderBatchMaxRows / AdaptiveBlockSizePredictor / EstimateCollectedEnough / CollectedEnoughWithColumns / BlockReaderByteBudget / SegmentColumnRawDataBytes / OrcReader / TableFormatReader / ProfileSpec / LocalExchanger) - Behavior changed: No - Does this need documentation: No Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
mrhhsg
commented
Apr 27, 2026
MemberAuthor
run buildall |
hello-stephen
commented
Apr 27, 2026
Contributor
BE UT Coverage ReportIncrement line coverage Increment coverage report
|
hello-stephen
commented
Apr 27, 2026
Contributor
FE UT Coverage ReportIncrement line coverage |
hello-stephen
commented
Apr 27, 2026
Contributor
BE Regression && UT Coverage ReportIncrement line coverage Increment coverage report
|
hello-stephen
commented
Apr 27, 2026
Contributor
FE Regression Coverage ReportIncrement line coverage |
yiguolei
approved these changes
Apr 28, 2026
Contributor
PR approved by at least one committer and no changes requested. |
Contributor
PR approved by anyone and no changes requested. |
Uh oh!
There was an error while loading. Please reload this page.
29 tasks
mrhhsg added a commit
to mrhhsg/doris
that referenced
this pull request
May 6, 2026
Issue Number: None Related PR: None Problem Summary: Add adaptive block row prediction for SegmentIterator, OLAP scan, file scan, and format readers. The scan path now uses a row ceiling plus preferred output byte budget to reduce oversized blocks for wide rows while preserving row-limited behavior for narrow rows. This commit also introduces the shared session/config/thrift/runtime budget plumbing used by later operators. Adds adaptive batch size controls for scan output blocks: preferred_block_size_bytes and preferred_max_column_in_block_size_bytes. - Test: Unit Test - Unit Test: ./run-be-ut.sh --run --filter=BlockBudgetTest.*:RuntimeStateBatchSizeTest.*:RuntimeStateBlockSizeBytesTest.*:RuntimeStateMaxColBytesTest.*:MockRuntimeStateBlockBudgetTest.*:AdaptiveBlockSizePredictorTest.*:BlockReaderBatchMaxRowsTest.*:EstimateCollectedEnoughTest.*:CollectedEnoughWithColumnsTest.*:BlockReaderByteBudgetTest.*:SegmentColumnRawDataBytesTest.*:CsvReaderSetBatchSizeTest.*:NewJsonReaderSetBatchSizeTest.*:OrcReaderTest.*:TableFormatReaderTest.*:ProfileSpecTest.*:LocalExchangerTest.* - Behavior changed: Yes (scan output block sizing can now be byte-budget limited when adaptive batch size is enabled) - Does this need documentation: Yes Issue Number: close #xxx Related PR: #xxx Problem Summary: None - Test <!-- At least one of them must be included. --> - [ ] Regression test - [ ] Unit Test - [ ] Manual test (add detailed scripts or steps below) - [ ] No need to test or manual test. Explain why: - [ ] This is a refactor/code format and no logic has been changed. - [ ] Previous test can cover this change. - [ ] No code files have been changed. - [ ] Other reason <!-- Add your reason? --> - Behavior changed: - [ ] No. - [ ] Yes. <!-- Explain the behavior change --> - Does this need documentation? - [ ] No. - [ ] Yes. <!-- Add document PR link here. eg: apache/doris-website#1214 --> - [ ] Confirm the release note - [ ] Confirm test cases - [ ] Confirm document - [ ] Add branch pick label <!-- Add branch pick label that this PR should merge into --> --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
mrhhsg added a commit
to mrhhsg/doris
that referenced
this pull request
May 10, 2026
Issue Number: None Related PR: None Problem Summary: Add adaptive block row prediction for SegmentIterator, OLAP scan, file scan, and format readers. The scan path now uses a row ceiling plus preferred output byte budget to reduce oversized blocks for wide rows while preserving row-limited behavior for narrow rows. This commit also introduces the shared session/config/thrift/runtime budget plumbing used by later operators. Adds adaptive batch size controls for scan output blocks: preferred_block_size_bytes and preferred_max_column_in_block_size_bytes. - Test: Unit Test - Unit Test: ./run-be-ut.sh --run --filter=BlockBudgetTest.*:RuntimeStateBatchSizeTest.*:RuntimeStateBlockSizeBytesTest.*:RuntimeStateMaxColBytesTest.*:MockRuntimeStateBlockBudgetTest.*:AdaptiveBlockSizePredictorTest.*:BlockReaderBatchMaxRowsTest.*:EstimateCollectedEnoughTest.*:CollectedEnoughWithColumnsTest.*:BlockReaderByteBudgetTest.*:SegmentColumnRawDataBytesTest.*:CsvReaderSetBatchSizeTest.*:NewJsonReaderSetBatchSizeTest.*:OrcReaderTest.*:TableFormatReaderTest.*:ProfileSpecTest.*:LocalExchangerTest.* - Behavior changed: Yes (scan output block sizing can now be byte-budget limited when adaptive batch size is enabled) - Does this need documentation: Yes Issue Number: close #xxx Related PR: #xxx Problem Summary: None - Test <!-- At least one of them must be included. --> - [ ] Regression test - [ ] Unit Test - [ ] Manual test (add detailed scripts or steps below) - [ ] No need to test or manual test. Explain why: - [ ] This is a refactor/code format and no logic has been changed. - [ ] Previous test can cover this change. - [ ] No code files have been changed. - [ ] Other reason <!-- Add your reason? --> - Behavior changed: - [ ] No. - [ ] Yes. <!-- Explain the behavior change --> - Does this need documentation? - [ ] No. - [ ] Yes. <!-- Add document PR link here. eg: apache/doris-website#1214 --> - [ ] Confirm the release note - [ ] Confirm test cases - [ ] Confirm document - [ ] Add branch pick label <!-- Add branch pick label that this PR should merge into --> --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
yiguolei pushed a commit
that referenced
this pull request
May 11, 2026
Pick PR: #62835 Problem Summary: Add adaptive block row prediction for SegmentIterator, OLAP scan, file scan, and format readers. The scan path now uses a row ceiling plus preferred output byte budget to reduce oversized blocks for wide rows while preserving row-limited behavior for narrow rows. This commit also introduces the shared session/config/thrift/runtime budget plumbing used by later operators. Adds adaptive batch size controls for scan output blocks: preferred_block_size_bytes and preferred_max_column_in_block_size_bytes. - Test: Unit Test - Unit Test: ./run-be-ut.sh --run --filter=BlockBudgetTest.*:RuntimeStateBatchSizeTest.*:RuntimeStateBlockSizeBytesTest.*:RuntimeStateMaxColBytesTest.*:MockRuntimeStateBlockBudgetTest.*:AdaptiveBlockSizePredictorTest.*:BlockReaderBatchMaxRowsTest.*:EstimateCollectedEnoughTest.*:CollectedEnoughWithColumnsTest.*:BlockReaderByteBudgetTest.*:SegmentColumnRawDataBytesTest.*:CsvReaderSetBatchSizeTest.*:NewJsonReaderSetBatchSizeTest.*:OrcReaderTest.*:TableFormatReaderTest.*:ProfileSpecTest.*:LocalExchangerTest.* - Behavior changed: Yes (scan output block sizing can now be byte-budget limited when adaptive batch size is enabled) - Does this need documentation: Yes Issue Number: close #xxx Related PR: #xxx Problem Summary: None - Test <!-- At least one of them must be included. --> - [ ] Regression test - [ ] Unit Test - [ ] Manual test (add detailed scripts or steps below) - [ ] No need to test or manual test. Explain why: - [ ] This is a refactor/code format and no logic has been changed. - [ ] Previous test can cover this change. - [ ] No code files have been changed. - [ ] Other reason <!-- Add your reason? --> - Behavior changed: - [ ] No. - [ ] Yes. <!-- Explain the behavior change --> - Does this need documentation? - [ ] No. - [ ] Yes. <!-- Add document PR link here. eg: apache/doris-website#1214 --> - [ ] Confirm the release note - [ ] Confirm test cases - [ ] Confirm document - [ ] Add branch pick label <!-- Add branch pick label that this PR should merge into --> --------- ### What problem does this PR solve? Issue Number: close #xxx Related PR: #xxx Problem Summary: ### Release note None ### Check List (For Author) - Test <!-- At least one of them must be included. --> - [ ] Regression test - [ ] Unit Test - [ ] Manual test (add detailed scripts or steps below) - [ ] No need to test or manual test. Explain why: - [ ] This is a refactor/code format and no logic has been changed. - [ ] Previous test can cover this change. - [ ] No code files have been changed. - [ ] Other reason <!-- Add your reason? --> - Behavior changed: - [ ] No. - [ ] Yes. <!-- Explain the behavior change --> - Does this need documentation? - [ ] No. - [ ] Yes. <!-- Add document PR link here. eg: apache/doris-website#1214 --> ### Check List (For Reviewer who merge this PR) - [ ] Confirm the release note - [ ] Confirm test cases - [ ] Confirm document - [ ] Add branch pick label <!-- Add branch pick label that this PR should merge into --> --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Closed
zhaorongsheng pushed a commit
to zhaorongsheng/doris
that referenced
this pull request
Jun 4, 2026
Issue Number: None Related PR: None Problem Summary: Add adaptive block row prediction for SegmentIterator, OLAP scan, file scan, and format readers. The scan path now uses a row ceiling plus preferred output byte budget to reduce oversized blocks for wide rows while preserving row-limited behavior for narrow rows. This commit also introduces the shared session/config/thrift/runtime budget plumbing used by later operators. Adds adaptive batch size controls for scan output blocks: preferred_block_size_bytes and preferred_max_column_in_block_size_bytes. - Test: Unit Test - Unit Test: ./run-be-ut.sh --run --filter=BlockBudgetTest.*:RuntimeStateBatchSizeTest.*:RuntimeStateBlockSizeBytesTest.*:RuntimeStateMaxColBytesTest.*:MockRuntimeStateBlockBudgetTest.*:AdaptiveBlockSizePredictorTest.*:BlockReaderBatchMaxRowsTest.*:EstimateCollectedEnoughTest.*:CollectedEnoughWithColumnsTest.*:BlockReaderByteBudgetTest.*:SegmentColumnRawDataBytesTest.*:CsvReaderSetBatchSizeTest.*:NewJsonReaderSetBatchSizeTest.*:OrcReaderTest.*:TableFormatReaderTest.*:ProfileSpecTest.*:LocalExchangerTest.* - Behavior changed: Yes (scan output block sizing can now be byte-budget limited when adaptive batch size is enabled) - Does this need documentation: Yes ### What problem does this PR solve? Issue Number: close #xxx Related PR: #xxx Problem Summary: ### Release note None ### Check List (For Author) - Test <!-- At least one of them must be included. --> - [ ] Regression test - [ ] Unit Test - [ ] Manual test (add detailed scripts or steps below) - [ ] No need to test or manual test. Explain why: - [ ] This is a refactor/code format and no logic has been changed. - [ ] Previous test can cover this change. - [ ] No code files have been changed. - [ ] Other reason <!-- Add your reason? --> - Behavior changed: - [ ] No. - [ ] Yes. <!-- Explain the behavior change --> - Does this need documentation? - [ ] No. - [ ] Yes. <!-- Add document PR link here. eg: apache/doris-website#1214 --> ### Check List (For Reviewer who merge this PR) - [ ] Confirm the release note - [ ] Confirm test cases - [ ] Confirm document - [ ] Add branch pick label <!-- Add branch pick label that this PR should merge into --> --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
16 tasks
mrhhsg added a commit
to mrhhsg/doris
that referenced
this pull request
Jul 28, 2026
### What problem does this PR solve? Issue Number: None Related PR: apache#62835 Problem Summary: AdaptiveBlockSizePredictor is shared by segment scans and external file scanners, but its declaration and implementation were owned by storage/segment. Move the predictor and its focused unit test to core/block, update all consumers, and remove unused segment-only metadata declarations so the core header no longer depends on storage. ### Release note None ### Check List (For Author) - Test: - Unit Test: ASAN focused harness compiled the unchanged production and test sources directly (25 tests passed) - Manual test: targeted ASAN_UT compilation of the predictor, test, and all three consumers; clang-format, check-format, changed-line clang-tidy, and diff check - Standard `run-be-ut.sh` did not complete because this worktree's third-party installation is missing the unrelated `liblance_c.a` - Behavior changed: No - Does this need documentation: No
mrhhsg added a commit
that referenced
this pull request
Jul 31, 2026
### What problem does this PR solve? Issue Number: None Related PR: #62835 Problem Summary: `AdaptiveBlockSizePredictor` is a block-level helper shared by segment scans and external file scanners. Keeping it under `storage/segment` forces external scan code to depend on the storage layer even though the predictor only uses `Block` statistics and EWMA calculations. This PR moves the predictor implementation and its focused unit test to `core/block`, updates all consumers to use the new include path, and removes the unused segment-specific `ColumnMetadata` declaration. It also makes `predict_next_rows()` const because it does not modify predictor state. The prediction behavior is unchanged. The branch has been rebuilt on the current `master`, preserving the later external-scan rollback from #65998 rather than restoring the old `update(rows, bytes)` path. ### Release note None ### Check List (For Author) - Test <!-- At least one of them must be included. --> - [ ] Regression test - [x] Unit Test - ASAN focused harness compiled the unchanged production and test sources directly: `AdaptiveBlockSizePredictorTest.*` (25 tests passed) - The standard `./run-be-ut.sh --run --filter='AdaptiveBlockSizePredictorTest.*' -j 48` runner did not complete because this worktree's third-party installation is missing the unrelated `liblance_c.a`; CI will provide the full monolithic BE validation - [x] Manual test (add detailed scripts or steps below) - Targeted ASAN_UT compilation passed for the predictor implementation, its focused test, and all three production consumers (`SegmentIterator`, `FileScanner`, and `FileScannerV2`) - `./build-support/clang-format.sh` - `./build-support/check-format.sh` - changed-line clang-tidy passed; the stock local invocation additionally encounters a pre-existing unmatched `NOLINTEND` in `be/src/core/types.h` - `git diff --check origin/master...HEAD` - [ ] No need to test or manual test. Explain why: - [ ] This is a refactor/code format and no logic has been changed. - [ ] Previous test can cover this change. - [ ] No code files have been changed. - [ ] Other reason <!-- Add your reason? --> - Behavior changed: - [x] No. - [ ] Yes. <!-- Explain the behavior change --> - Does this need documentation? - [x] No. - [ ] Yes. <!-- Add document PR link here. eg: apache/doris-website#1214 --> ### Check List (For Reviewer who merge this PR) - [ ] Confirm the release note - [ ] Confirm test cases - [ ] Confirm document - [ ] Add branch pick label <!-- Add branch pick label that this PR should merge into -->
wyxxxcat pushed a commit
to wyxxxcat/doris
that referenced
this pull request
Aug 17, 2026
### What problem does this PR solve? Issue Number: None Related PR: apache#62835 Problem Summary: `AdaptiveBlockSizePredictor` is a block-level helper shared by segment scans and external file scanners. Keeping it under `storage/segment` forces external scan code to depend on the storage layer even though the predictor only uses `Block` statistics and EWMA calculations. This PR moves the predictor implementation and its focused unit test to `core/block`, updates all consumers to use the new include path, and removes the unused segment-specific `ColumnMetadata` declaration. It also makes `predict_next_rows()` const because it does not modify predictor state. The prediction behavior is unchanged. The branch has been rebuilt on the current `master`, preserving the later external-scan rollback from apache#65998 rather than restoring the old `update(rows, bytes)` path. ### Release note None ### Check List (For Author) - Test <!-- At least one of them must be included. --> - [ ] Regression test - [x] Unit Test - ASAN focused harness compiled the unchanged production and test sources directly: `AdaptiveBlockSizePredictorTest.*` (25 tests passed) - The standard `./run-be-ut.sh --run --filter='AdaptiveBlockSizePredictorTest.*' -j 48` runner did not complete because this worktree's third-party installation is missing the unrelated `liblance_c.a`; CI will provide the full monolithic BE validation - [x] Manual test (add detailed scripts or steps below) - Targeted ASAN_UT compilation passed for the predictor implementation, its focused test, and all three production consumers (`SegmentIterator`, `FileScanner`, and `FileScannerV2`) - `./build-support/clang-format.sh` - `./build-support/check-format.sh` - changed-line clang-tidy passed; the stock local invocation additionally encounters a pre-existing unmatched `NOLINTEND` in `be/src/core/types.h` - `git diff --check origin/master...HEAD` - [ ] No need to test or manual test. Explain why: - [ ] This is a refactor/code format and no logic has been changed. - [ ] Previous test can cover this change. - [ ] No code files have been changed. - [ ] Other reason <!-- Add your reason? --> - Behavior changed: - [x] No. - [ ] Yes. <!-- Explain the behavior change --> - Does this need documentation? - [x] No. - [ ] Yes. <!-- Add document PR link here. eg: apache/doris-website#1214 --> ### Check List (For Reviewer who merge this PR) - [ ] Confirm the release note - [ ] Confirm test cases - [ ] Confirm document - [ ] Add branch pick label <!-- Add branch pick label that this PR should merge into -->
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Issue Number: None
Related PR: None
Problem Summary: Add adaptive block row prediction for SegmentIterator, OLAP scan, file scan, and format readers. The scan path now uses a row ceiling plus preferred output byte budget to reduce oversized blocks for wide rows while preserving row-limited behavior for narrow rows. This commit also introduces the shared session/config/thrift/runtime budget plumbing used by later operators.
Adds adaptive batch size controls for scan output blocks: preferred_block_size_bytes and preferred_max_column_in_block_size_bytes.
What problem does this PR solve?
Issue Number: close #xxx
Related PR: #xxx
Problem Summary:
Release note
None
Check List (For Author)
Test
Behavior changed:
Does this need documentation?
Check List (For Reviewer who merge this PR)