Uh oh!
There was an error while loading. Please reload this page.
[fix](memory) track IO layer read buffers via MemTrackerLimiter using PODArray - #62032
Merged
Merged
Conversation
Thearas
commented
Apr 2, 2026
Contributor
Thank you for your contribution to Apache Doris. Please clearly describe your PR:
|
sollhuiforce-pushed
the
fix_io_memory_tracker
branch
from
April 2, 2026 03:35
994e1b2 to
d65f836Comparesollhui
commented
Apr 2, 2026
ContributorAuthor
run buildall |
sollhuiforce-pushed
the
fix_io_memory_tracker
branch
from
April 2, 2026 04:57
d65f836 to
4e9989bComparesollhui
commented
Apr 2, 2026
ContributorAuthor
run buildall |
doris-robot
commented
Apr 2, 2026
BE UT Coverage ReportIncrement line coverage Increment coverage report
|
doris-robot
commented
Apr 2, 2026
TPC-H: Total hot run time: 29179 ms |
doris-robot
commented
Apr 2, 2026
TPC-DS: Total hot run time: 178890 ms |
sollhuiforce-pushed
the
fix_io_memory_tracker
branch
from
April 2, 2026 07:00
4e9989b to
7747afeComparesollhui
commented
Apr 2, 2026
ContributorAuthor
run buildall |
doris-robot
commented
Apr 2, 2026
TPC-H: Total hot run time: 29396 ms |
doris-robot
commented
Apr 2, 2026
TPC-DS: Total hot run time: 179812 ms |
hello-stephen
commented
Apr 2, 2026
Contributor
BE UT Coverage ReportIncrement line coverage Increment coverage report
|
sollhuiforce-pushed
the
fix_io_memory_tracker
branch
from
April 2, 2026 08:52
7747afe to
35bb6d5Comparesollhui
commented
Apr 2, 2026
ContributorAuthor
run buildall |
doris-robot
commented
Apr 2, 2026
BE UT Coverage ReportIncrement line coverage Increment coverage report
|
yiguolei
approved these changes
Apr 2, 2026
Contributor
PR approved by at least one committer and no changes requested. |
Contributor
PR approved by anyone and no changes requested. |
yiguolei
approved these changes
Apr 2, 2026
doris-robot
commented
Apr 2, 2026
TPC-H: Total hot run time: 29162 ms |
doris-robot
commented
Apr 2, 2026
TPC-DS: Total hot run time: 180691 ms |
Uh oh!
There was an error while loading. Please reload this page.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Several IO layer read buffers were allocated via
std::make_unique<char[]>/new char[], which bypasses Doris's memory tracking system (MemTrackerLimiter).These allocations are invisible to the query memory tracker, leading to
under-reported memory usage and potential OOM surprises under concurrent S3/HDFS
scans.
Affected locations:
PrefetchBuffer::_bufinbuffered_reader— the main S3/HDFS prefetch bufferHttpFileReader::_read_buffer— the 1 MB HTTP read bufferHdfsFileSystem::download_impl— the 1 MB copy buffer for remote→local downloadSolution
Replace raw
unique_ptr<char[]>allocations withPODArray<char>, which usesDoris's
Allocator<..., check_and_tracking_memory=true>internally. Everyalloc/realloc/freecall goes throughconsume_memory/release_memoryon the thread-local
MemTrackerLimiter, so these buffers are now properlyaccounted for.
Key behavioral notes:
PODArraydefault constructor allocates no memory, so the lazy-allocationoptimization in
PrefetchBuffer(introduced to reduce peak memory during TVFscans over many small S3 files) is fully preserved — the buffer is only
allocated on the first actual prefetch call.
PODArraysupports move semantics, so the existing move constructor ofPrefetchBufferworks unchanged.Changes
buffered_reader.h_buf:unique_ptr<char[]>→PODArray<char>; remove eagernew char[]from constructorbuffered_reader.cpp_buf.empty()/_buf.resize);.get()→.data()forPrefetchBufferhttp_file_reader.h_read_buffer:unique_ptr<char[]>→PODArray<char>http_file_reader.cppmake_unique<char[]>→resize;.get()→.data()hdfs_file_system.cppnew char[]→PODArray<char>