Uh oh!
There was an error while loading. Please reload this page.
[feature](file cache) Support in-memory filecache for no-disk/slow-disk system - #38811
Conversation
doris-robot
commented
Aug 2, 2024
Thank you for your contribution to Apache Doris. Since 2024-03-18, the Document has been moved to doris-website. |
| namespace doris::io { | ||
| MemFileCacheStorage::~MemFileCacheStorage() {} |
There was a problem hiding this comment.
warning: use '= default' to define a trivial destructor [modernize-use-equals-default]
| MemFileCacheStorage::~MemFileCacheStorage() {} | |
| MemFileCacheStorage::~MemFileCacheStorage() = default; |
| namespace doris::io { | ||
| struct MemBlock { | ||
| std::shared_ptr<char> addr; |
There was a problem hiding this comment.
It is a memory address pointing to a char array (a continous memory block), so I think it should be std::shared_ptr<char[]> instead?
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
freemandealer
commented
Aug 29, 2024
TODO: config: for extensibility, use another json field instead of 'memory' in path |
Sry for this hanging WIP PR, I've been dealing with more urgent tasks. I will return to this ASAP! |
| LOG_ERROR("").tag("file cache path", cache_base_path).tag("error", strerror(errno)); | ||
| return Status::IOError("{} statfs error {}", cache_base_path, strerror(errno)); | ||
| } | ||
| size_t disk_capacity = static_cast<size_t>( |
There was a problem hiding this comment.
warning: use auto when initializing with a cast to avoid duplicating the type name [modernize-use-auto]
| size_t disk_capacity = static_cast<size_t>( | |
| auto disk_capacity = static_cast<size_t>( |
| } | ||
| } | ||
| void test_file_cache_memory_storage(io::FileCacheType cache_type) { |
There was a problem hiding this comment.
warning: function 'test_file_cache_memory_storage' exceeds recommended size/complexity thresholds [readability-function-size]
voidtest_file_cache_memory_storage(io::FileCacheType cache_type) {
^Additional context
be/test/io/cache/block_file_cache_test.cpp:659: 397 lines including whitespace and comments (threshold 80)
voidtest_file_cache_memory_storage(io::FileCacheType cache_type) {
^| TEST_F(BlockFileCacheTest, ttl_modify_memory_storage) { | ||
| test_file_cache_memory_storage(io::FileCacheType::NORMAL); | ||
| auto sp = SyncPoint::get_instance(); |
There was a problem hiding this comment.
warning: 'auto sp' can be declared as 'auto *sp' [readability-qualified-auto]
| auto sp = SyncPoint::get_instance(); | |
| L);auto * |
| } | ||
| } | ||
| void test_file_cache_memory_storage(io::FileCacheType cache_type) { |
There was a problem hiding this comment.
warning: function 'test_file_cache_memory_storage' exceeds recommended size/complexity thresholds [readability-function-size]
voidtest_file_cache_memory_storage(io::FileCacheType cache_type) {
^Additional context
be/test/io/cache/block_file_cache_test.cpp:659: 389 lines including whitespace and comments (threshold 80)
voidtest_file_cache_memory_storage(io::FileCacheType cache_type) {
^freemandealer
commented
Sep 6, 2024
run buildall |
1 similar comment
freemandealer
commented
Sep 6, 2024
run buildall |
doris-robot
commented
Sep 6, 2024
TPC-H: Total hot run time: 38581 ms |
doris-robot
commented
Sep 6, 2024
TPC-DS: Total hot run time: 187710 ms |
doris-robot
commented
Sep 6, 2024
ClickBench: Total hot run time: 31.52 s |
doris-robot
commented
Sep 6, 2024
TeamCity be ut coverage result: |
a70c120 to
342316bComparefreemandealer
commented
Sep 23, 2024
run buildall |
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
doris-robot
commented
Sep 23, 2024
TPC-H: Total hot run time: 41403 ms |
freemandealer
commented
Sep 23, 2024
run buildall |
PR approved by at least one committer and no changes requested. |
doris-robot
commented
Sep 23, 2024
ClickBench: Total hot run time: 32.17 s |
TODO: 1. 优化 内存池 2. 写入、读取数据内存零拷贝 3. 原始一些代码的重构(很多fs相关的内容散布在 block_file_cache.cpp 里, 需要进行额外的清理) Signed-off-by: freemandealer <freeman.zhang1992@gmail.com>
Signed-off-by: freemandealer <freeman.zhang1992@gmail.com>
Signed-off-by: freemandealer <freeman.zhang1992@gmail.com>
Signed-off-by: freemandealer <freeman.zhang1992@gmail.com>
Signed-off-by: freemandealer <freeman.zhang1992@gmail.com>
Signed-off-by: freemandealer <freeman.zhang1992@gmail.com>
2b80049 to
b78918aComparefreemandealer
commented
Sep 24, 2024
run buildall |
PR approved by at least one committer and no changes requested. |
doris-robot
commented
Sep 24, 2024
TeamCity be ut coverage result: |
doris-robot
commented
Sep 24, 2024
TPC-H: Total hot run time: 41701 ms |
doris-robot
commented
Sep 24, 2024
TPC-DS: Total hot run time: 191397 ms |
doris-robot
commented
Sep 24, 2024
ClickBench: Total hot run time: 32.42 s |
…sk system (#38811) User can set file_cache_path=[{..., "storage":"memory"}] to use RAM as file cache storage.
…sk system (apache#38811) User can set file_cache_path=[{..., "storage":"memory"}] to use RAM as file cache storage.
User can set file_cache_path=[{..., "storage":"memory"}] to use RAM as file cache storage.