[improve](cache) File cache async init - #39036

Merged

morningman merged 9 commits into

apache:branch-2.1from

suxiaogang223:file_cache_async_init

Aug 15, 2024

suxiaogang223 commented Aug 7, 2024

Member

Proposed changes

Do load_cache_info_into_memory() asynchronously in a background thread in LRUFileCache::initialize().
When the cache is not ready, LRUFileCache::get_or_set() will return the FileBlock which state is SKIP_CACHE.

suxiaogang223 added 2 commits

August 6, 2024 18:11

async load_cache_info_into_memory in LRUFileCache::initialize()

55a3694

get_or_set will return the FileBlock which state is SKIP_CACHE

ab293ad

doris-robot commented Aug 7, 2024

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

github-actionsBot reviewed

View reviewed changes

github-actionsBot left a comment

Contributor

clang-tidy made some suggestions

be/src/io/cache/block/block_lru_file_cache.cpp Outdated

be/src/io/cache/block/block_lru_file_cache.cpp Outdated

add log

062a32e

github-actionsBot reviewed

View reviewed changes

github-actionsBot left a comment

Contributor

clang-tidy made some suggestions

be/src/io/cache/block/block_lru_file_cache.cpp Outdated

be/src/io/cache/block/block_lru_file_cache.cpp Outdated

revert load_cache_info_into_memory

a1d0f43

github-actionsBot commented Aug 7, 2024

Contributor

clang-tidy review says "All clean, LGTM! 👍"

morningman added the dev/2.1.x label

fix

f2b118a

github-actionsBot commented Aug 8, 2024

Contributor

clang-tidy review says "All clean, LGTM! 👍"

suxiaogang223 commented Aug 8, 2024

MemberAuthor

run buildall

add scan_file_num_to_sleep and sleep_time_per_scan_file to limit IO

47224ea

github-actionsBot commented Aug 9, 2024

Contributor

clang-tidy review says "All clean, LGTM! 👍"

suxiaogang223 commented Aug 9, 2024

MemberAuthor

run buildall

rename

be456e1

github-actionsBot commented Aug 9, 2024

Contributor

clang-tidy review says "All clean, LGTM! 👍"

morningman reviewed

View reviewed changes

be/src/io/cache/block/block_lru_file_cache.cpp Outdated

use vlog

b517b10

suxiaogang223 commented Aug 13, 2024

MemberAuthor

run buildall

github-actionsBot commented Aug 13, 2024

Contributor

clang-tidy review says "All clean, LGTM! 👍"

use fmt::format instead of std::format

70a0514

suxiaogang223 commented Aug 14, 2024

MemberAuthor

run buildall

github-actionsBot commented Aug 14, 2024

Contributor

clang-tidy review says "All clean, LGTM! 👍"

morningman merged commit 0680c8d into apache:branch-2.1

morningman added dev/2.1.6-merged and removed dev/2.1.x labels

suxiaogang223 deleted the file_cache_async_init branch

August 15, 2024 08:47

morningman pushed a commit to morningman/doris that referenced this pull request

[improve](cache) File cache async initapache#39036

12f0411

yiguolei mentioned this pull request

Release Note 2.1.6 #40443

Closed

3 tasks

cambyzju mentioned this pull request

[fix](crash) be crash on ~LRUFileCache #42498

Merged

yiguolei pushed a commit that referenced this pull request

[fix](crash) be crash on ~LRUFileCache (#42498)

aa00533

come from: #39036
Crash stack:
```
(gdb) bt
#0 0x00007ff5c8c6c387 in raise () from /lib64/libc.so.6
#1 0x00007ff5c8c6da78 in abort () from /lib64/libc.so.6
#2 0x0000561eb0a5e38a in __gnu_cxx::__verbose_terminate_handler () at ../../../../libstdc++-v3/libsupc++/vterminate.cc:95
#3 0x0000561eb0a5caf6 in __cxxabiv1::__terminate (handler=<optimized out>) at ../../../../libstdc++-v3/libsupc++/eh_terminate.cc:48
#4 0x0000561eb0a5cb61 in std::terminate () at ../../../../libstdc++-v3/libsupc++/eh_terminate.cc:58
#5 0x0000561ea4028552 in doris::io::LRUFileCache::~LRUFileCache (this=0x7ff540fed000) at /root/be/src/io/cache/block/block_lru_file_cache.h:62
#6 0x0000561ea402857e in doris::io::LRUFileCache::~LRUFileCache (this=0x33688) at /root/be/src/io/cache/block/block_lru_file_cache.h:54
#7 0x0000561ea4251cd2 in std::default_delete<doris::io::IFileCache>::operator() (this=0x7ff54101d000, __ptr=0x33688)
at /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/unique_ptr.h:85
#8 std::unique_ptr<doris::io::IFileCache, std::default_delete<doris::io::IFileCache> >::~unique_ptr (this=0x7ff54101d000)
at /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/unique_ptr.h:361
#9 std::destroy_at<std::unique_ptr<doris::io::IFileCache, std::default_delete<doris::io::IFileCache> > > (__location=0x7ff54101d000)
at /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/stl_construct.h:88
#10 std::_Destroy<std::unique_ptr<doris::io::IFileCache, std::default_delete<doris::io::IFileCache> > > (__pointer=0x7ff54101d000)
at /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/stl_construct.h:138
#11 std::_Destroy_aux<false>::__destroy<std::unique_ptr<doris::io::IFileCache, std::default_delete<doris::io::IFileCache> >*> (__first=0x7ff54101d000, __last=0x7ff54101d008) at /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/stl_construct.h:152
#12 std::_Destroy<std::unique_ptr<doris::io::IFileCache, std::default_delete<doris::io::IFileCache> >*> (__first=<optimized out>, __last=0x7ff54101d008)
at /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/stl_construct.h:184
#13 std::_Destroy<std::unique_ptr<doris::io::IFileCache, std::default_delete<doris::io::IFileCache> >*, std::unique_ptr<doris::io::IFileCache, std::default_delete<doris::io::IFileCache> > > (__first=<optimized out>, __last=0x7ff54101d008)
at /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/alloc_traits.h:746
#14 std::vector<std::unique_ptr<doris::io::IFileCache, std::default_delete<doris::io::IFileCache> >, std::allocator<std::unique_ptr<doris::io::IFileCache, std::default_delete<doris::io::IFileCache> > > >::~vector (this=0x7ff596a24ba8)
at /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/stl_vector.h:680
#15 doris::io::FileCacheFactory::~FileCacheFactory (this=0x7ff596a24b80) at /root/be/src/io/cache/block/block_file_cache_factory.h:42
#16 doris::ExecEnv::destroy (this=0x561eb130b800 <doris::ExecEnv::GetInstance()::s_exec_env>) at /root/be/src/runtime/exec_env_init.cpp:651
#17 0x0000561ea35c3fe3 in main (argc=<optimized out>, argv=<optimized out>) at /root/be/src/service/doris_main.cpp:628
```

Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment