Skip to content

Add lz4 and zstd compression support to the CLFUS RAM cache - #13257

Open
phongn wants to merge 13 commits into
apache:masterfrom
phongn:lz4
Open

phongn wants to merge 13 commits into
apache:masterfrom
phongn:lz4

Conversation

@phongn

@phongn phongn commented Jun 11, 2026

Copy link
Copy Markdown
Collaborator

Add lz4 and zstd compression support to the CLFUS RAM cache

Summary

This adds two modern compression backends to the CLFUS RAM cache, as optional build-time dependencies:

  • lz4 (proxy.config.cache.ram_cache.compress: 4) — a replacement for fastlz: strictly better compression ratio at substantially higher throughput.
  • zstd (proxy.config.cache.ram_cache.compress: 5, level 3) — a replacement for libz/deflate-6: comparable ratio at roughly 10× the compression speed and 3× the decompression speed.

The existing fastlz/libz/liblzma backends are unchanged and remain valid for their config values; the docs now recommend lz4 over fastlz and zstd over libz.

Benchmarks (lzbench, silesia XML corpus, Xeon Gold 6338, one thread):

Method Compress Decompress Final size
fastlz 452 MB/s 913 MB/s ~26%
lz4 727 MB/s 3458 MB/s ~23%
libz-6 54 MB/s 536 MB/s ~13%
zstd-3 508 MB/s 1690 MB/s ~12%

Implementation notes

  • lz4 uses the one-shot LZ4_compress_default / LZ4_decompress_safe API.
  • zstd uses per-thread reusable contexts (thread_local ZSTD_CCtx/ZSTD_DCtx) with ZSTD_compress2 and a sticky compression level, avoiding a context allocation per call — relevant since decompression sits on the cache-hit path. This requires zstd ≥ 1.4.0 (the first release with the advanced one-shot API stable); the version floor is enforced in find_package.
  • Misconfiguration is caught at startup: configuring a backend that wasn't compiled in is Fatal, matching the existing liblzma behavior.
  • New cmake/FindLZ4.cmake and cmake/FindZSTD.cmake modules, written in the same shape as the tree's other Find*.cmake (Apache header, find_library/find_path, INTERFACE IMPORTED target) plus the header-version parse needed to enforce a floor. zstd detection previously used find_package(zstd CONFIG) only, which fails on distributions that don't ship a cmake config package; it now resolves via the module.
  • The zstd::zstd alias shim is kept, and it is not dead code. With CMAKE_FIND_PACKAGE_PREFER_CONFIG the lookup can resolve through zstd's own config package rather than the module: CMake searches for ZSTDConfig.cmake, and on a case-insensitive filesystem that matches the zstdConfig.cmake zstd installs. That package exports zstd::libzstd_shared/_static, so without the alias the four targets linking zstd::zstd fail at generate time on macOS. The shim now also disables zstd with a warning if the config package exports none of the names it knows, rather than failing the generate.
  • lz4 requires ≥ 1.7.5: 1.7.0 (r129) introduced LZ4_compress_default(), and LZ4_versionString(), which traffic_layout reports, needs 1.7.5.
  • traffic_layout info now reports TS_HAS_LZ4, and --versions reports the lz4 runtime version under lz4, matching what the existing zstd key means.

Testing

The RamCacheCLFUS class definition moved from the .cc into a new private header so unit tests can drive compress_entries() synchronously. A new Catch2 test (test_RamCacheCLFUS) does store → compress → read-back roundtrips across all five codecs plus the uncompressed case, asserting byte-for-byte equality and the expected RAM_HIT_COMPRESS_* state, plus incompressible-fallback and small-payload cases. No prior test verified compression data integrity for any backend.

Drive-by fixes

  • The liblzma compress path sized its output buffer at e->len instead of lzma_stream_buffer_bound(e->len); it now matches the other backends, with the existing REQUIRED_COMPRESSION/REQUIRED_SHRINK thresholds deciding what to keep. To be precise about what this fixes: it is a robustness fix, not a memory-safety or memory-saving one. lzma_easy_buffer_encode() is told the output size and returns LZMA_BUF_ERROR rather than overrunning, so the old code was never unsafe — it just failed the encode for any object that didn't shrink, and that failure marked the entry incompressible. Old code and new code both end up reading back as RAM_HIT_COMPRESS_NONE for such an object (the old one by failing, the new one by storing it raw), so no test can discriminate the two; what changes is that objects liblzma can compress but whose bound exceeds e->len now get compressed instead of being written off.
  • proxy.config.cache.ram_cache.compress had validity pattern [0-3] in RecordsConfig.cc, so RecYAMLDecoder rejected the new values 4 and 5 at load time, logged a validity warning and fell back to the default of 0. Without this, setting 4 or 5 in records.yaml as the docs describe would silently leave compression off. Widened to [0-5], with a unit test that walks every CACHE_COMPRESSION_* value against the record's own check and pattern so the range cannot drift behind the enum again.
  • Removed a stale unconditional "libz not available for RAM cache compression" warning that fired every second when compress: 2 was configured — zlib is a required dependency and always available.

Observability

Two new counters, global and per-volume:

  • ram_cache.decompress.failure — an entry failed to decompress on read. The entry is dropped and the read becomes a miss, and a throttled Warning carries the codec's own diagnosis (ZSTD_getErrorName(), zError(), or the codec's numeric return) so a nonzero counter can be told apart from a corrupt frame versus a bookkeeping error in e->len. Previously any decode failure was indistinguishable from an ordinary miss outside a debug build.
  • ram_cache.compress.failure — an entry the compression library could not compress. Objects that merely did not shrink enough are deliberately not counted; that is the ordinary outcome for already-compressed content and would swamp the signal. A thread whose zstd context cannot be allocated skips its compression pass entirely and is not counted here either, since that would make the counter climb once a second per stripe for the life of the process; the one-time Warning reports that condition.

CI / packaging

liblz4-dev / lz4-devel added to the deb and yum CI images. The Fedora CI image will need lz4-devel added for build coverage of the new backend (zstd-devel is already present).

Future work

Compression is currently implemented inside CLFUS only. A follow-up refactor will extract it into a shared layer so all RAM cache algorithms can use it; the new unit test is structured to migrate there.

@bryancall bryancall left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Took a pass over this — really nice change. The liblzma lzma_stream_buffer_bound() fix is a real correctness improvement, the per-thread zstd context design is clean, and adding compression-integrity tests where there were none is great. A handful of improvement suggestions inline; none are blockers.

The one I'd most encourage looking at is the sticky-null behavior of the thread-local zstd context: a one-time allocation failure silently degrades that thread to uncompressed forever (and evicts valid entries on read) with no log or metric. Details inline.

Comment thread src/iocore/cache/RamCacheCLFUS.cc
Comment thread src/iocore/cache/RamCacheCLFUS.cc
Comment thread src/iocore/cache/unit_tests/test_RamCacheCLFUS.cc Outdated
Comment thread include/iocore/cache/Cache.h
Comment thread doc/developer-guide/cache-architecture/ram-cache.en.rst Outdated
@cmcfarlen cmcfarlen added this to the 11.0.0 milestone Jun 15, 2026
@cmcfarlen
cmcfarlen self-requested a review June 15, 2026 22:35
@ezelkow1

ezelkow1 commented Jun 15, 2026

Copy link
Copy Markdown
Member

Since I know you probably cant get to the CI output @phongn , here's the failure. It's in the clfus catch test:

28/162 Test  #30: test_cache_RamCacheCLFUS ...............***Failed    0.87 sec
Randomness seeded to: 1464257562
[Jun 11 21:42:58.809] RamCacheCLFUS DIAG: <Stripe.cc:154 (_init_directory)> (cache_init) Stripe  0:10: allocating 24576 directory bytes for a 81920 byte volume (30.000000%)
[Jun 11 21:42:58.923] RamCacheCLFUS DIAG: <Stripe.cc:178 (~Stripe)> (cache_free) Stripe  0:10: freeing raw_dir=0x62c000009000 size=24576 huge=false
[Jun 11 21:42:58.926] RamCacheCLFUS DIAG: <Stripe.cc:154 (_init_directory)> (cache_init) Stripe  0:10: allocating 24576 directory bytes for a 81920 byte volume (30.000000%)
[Jun 11 21:42:58.932] RamCacheCLFUS DIAG: <Stripe.cc:178 (~Stripe)> (cache_free) Stripe  0:10: freeing raw_dir=0x62c000011000 size=24576 huge=false
[Jun 11 21:42:58.934] RamCacheCLFUS DIAG: <Stripe.cc:154 (_init_directory)> (cache_init) Stripe  0:10: allocating 24576 directory bytes for a 81920 byte volume (30.000000%)
[Jun 11 21:42:58.942] RamCacheCLFUS DIAG: <Stripe.cc:178 (~Stripe)> (cache_free) Stripe  0:10: freeing raw_dir=0x62c000019000 size=24576 huge=false
[Jun 11 21:42:58.944] RamCacheCLFUS DIAG: <Stripe.cc:154 (_init_directory)> (cache_init) Stripe  0:10: allocating 24576 directory bytes for a 81920 byte volume (30.000000%)
[Jun 11 21:42:58.965] RamCacheCLFUS DIAG: <Stripe.cc:178 (~Stripe)> (cache_free) Stripe  0:10: freeing raw_dir=0x62c000021000 size=24576 huge=false
[Jun 11 21:42:58.967] RamCacheCLFUS DIAG: <Stripe.cc:154 (_init_directory)> (cache_init) Stripe  0:10: allocating 24576 directory bytes for a 81920 byte volume (30.000000%)
[Jun 11 21:42:58.974] RamCacheCLFUS DIAG: <Stripe.cc:178 (~Stripe)> (cache_free) Stripe  0:10: freeing raw_dir=0x62c000031000 size=24576 huge=false
[Jun 11 21:42:58.977] RamCacheCLFUS DIAG: <Stripe.cc:154 (_init_directory)> (cache_init) Stripe  0:10: allocating 24576 directory bytes for a 81920 byte volume (30.000000%)
[Jun 11 21:42:58.987] RamCacheCLFUS DIAG: <Stripe.cc:178 (~Stripe)> (cache_free) Stripe  0:10: freeing raw_dir=0x62c000039000 size=24576 huge=false
[Jun 11 21:42:58.989] RamCacheCLFUS DIAG: <Stripe.cc:154 (_init_directory)> (cache_init) Stripe  0:10: allocating 24576 directory bytes for a 81920 byte volume (30.000000%)
[Jun 11 21:42:59.002] RamCacheCLFUS DIAG: <Stripe.cc:178 (~Stripe)> (cache_free) Stripe  0:10: freeing raw_dir=0x62c000041000 size=24576 huge=false
[Jun 11 21:42:59.004] RamCacheCLFUS DIAG: <Stripe.cc:154 (_init_directory)> (cache_init) Stripe  0:10: allocating 24576 directory bytes for a 81920 byte volume (30.000000%)
[Jun 11 21:42:59.121] RamCacheCLFUS DIAG: <Stripe.cc:178 (~Stripe)> (cache_free) Stripe  0:10: freeing raw_dir=0x62c000049000 size=24576 huge=false
[Jun 11 21:42:59.123] RamCacheCLFUS DIAG: <Stripe.cc:154 (_init_directory)> (cache_init) Stripe  0:10: allocating 24576 directory bytes for a 81920 byte volume (30.000000%)
[Jun 11 21:42:59.126] RamCacheCLFUS DIAG: <Stripe.cc:178 (~Stripe)> (cache_free) Stripe  0:10: freeing raw_dir=0x62c000051000 size=24576 huge=false
[Jun 11 21:42:59.129] RamCacheCLFUS DIAG: <Stripe.cc:154 (_init_directory)> (cache_init) Stripe  0:10: allocating 24576 directory bytes for a 81920 byte volume (30.000000%)
[Jun 11 21:42:59.129] RamCacheCLFUS DIAG: <Stripe.cc:178 (~Stripe)> (cache_free) Stripe  0:10: freeing raw_dir=0x62c000059000 size=24576 huge=false

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
RamCacheCLFUS is a Catch2 v3.9.1 host application.
Run with -? for options

-------------------------------------------------------------------------------
CLFUS compression backends compiled in
-------------------------------------------------------------------------------
../src/iocore/cache/unit_tests/test_RamCacheCLFUS.cc:253
...............................................................................

../src/iocore/cache/unit_tests/test_RamCacheCLFUS.cc:259: warning:
  lz4 is not compiled in; the lz4 RAM cache compression backend is NOT tested

===============================================================================
All tests passed (45 assertions in 4 test cases)

=================================================================
==7803==ERROR: AddressSanitizer: heap-use-after-free on address 0x62f0000004f8 at pc 0x00000049078a bp 0x7f263c3ddbd0 sp 0x7f263c3ddbc0
WRITE of size 8 at 0x62f0000004f8 thread T1 ([ET_NET 0])
    #0 0x490789 in std::__atomic_base<long>::fetch_add(long, std::memory_order) /opt/rh/gcc-toolset-11/root/usr/include/c++/11/bits/atomic_base.h:636
    #1 0x490789 in ts::Metrics::Counter::increment(ts::Metrics::Counter::AtomicType*, unsigned long) ../include/tsutil/Metrics.h:527
    #2 0x5c8e79 in NetHandler::waitForActivity(long) ../src/iocore/net/NetHandler.cc:350
    #3 0xd042e9 in EThread::execute_regular() ../src/iocore/eventsystem/UnixEThread.cc:326
    #4 0xd0488f in EThread::execute() ../src/iocore/eventsystem/UnixEThread.cc:383
    #5 0xd012fe in spawn_thread_internal ../src/iocore/eventsystem/Thread.cc:75
    #6 0x7f2640d421c9 in start_thread (/lib64/libpthread.so.0+0x81c9)
    #7 0x7f264099d952 in clone (/lib64/libc.so.6+0x39952)

0x62f0000004f8 is located 248 bytes inside of 49152-byte region [0x62f000000400,0x62f00000c400)
freed by thread T0 here:
    #0 0x7f264397236f in operator delete(void*, unsigned long) (/lib64/libasan.so.6+0xb736f)
    #1 0xdf3e0c in std::default_delete<std::tuple<std::array<std::tuple<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, int>, 1024ul>, std::array<ts::Metrics::AtomicType, 1024ul> > >::operator()(std::tuple<std::array<std::tuple<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, int>, 1024ul>, std::array<ts::Metrics::AtomicType, 1024ul> >*) const /opt/rh/gcc-toolset-11/root/usr/include/c++/11/bits/unique_ptr.h:85
    #2 0xdf2a8a in std::unique_ptr<std::tuple<std::array<std::tuple<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, int>, 1024ul>, std::array<ts::Metrics::AtomicType, 1024ul> >, std::default_delete<std::tuple<std::array<std::tuple<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, int>, 1024ul>, std::array<ts::Metrics::AtomicType, 1024ul> > > >::~unique_ptr() /opt/rh/gcc-toolset-11/root/usr/include/c++/11/bits/unique_ptr.h:361

=================================================================
    #3 0xdf1643 in std::array<std::unique_ptr<std::tuple<std::array<std::tuple<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, int>, 1024ul>, std::array<ts::Metrics::AtomicType, 1024ul> >, std::default_delete<std::tuple<std::array<std::tuple<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, int>, 1024ul>, std::array<ts::Metrics::AtomicType, 1024ul> > > >, 8192ul>::~array() /opt/rh/gcc-toolset-11/root/usr/include/c++/11/array:95
==7803==ERROR: LeakSanitizer: detected memory leaks
    #4 0xdf1d31 in ts::Metrics::Storage::~Storage() ../include/tsutil/Metrics.h:325

    #5 0xdfc516 in void std::destroy_at<ts::Metrics::Storage>(ts::Metrics::Storage*) /opt/rh/gcc-toolset-11/root/usr/include/c++/11/bits/stl_construct.h:88
Direct leak of 5080 byte(s) in 5 object(s) allocated from:
    #6 0xdfc4ed in void std::allocator_traits<std::allocator<ts::Metrics::Storage> >::destroy<ts::Metrics::Storage>(std::allocator<ts::Metrics::Storage>&, ts::Metrics::Storage*) /opt/rh/gcc-toolset-11/root/usr/include/c++/11/bits/alloc_traits.h:537
    #0 0x7f264396f9a7 in __interceptor_malloc (/lib64/libasan.so.6+0xb49a7)
    #7 0xdfc306 in std::_Sp_counted_ptr_inplace<ts::Metrics::Storage, std::allocator<ts::Metrics::Storage>, (__gnu_cxx::_Lock_policy)2>::_M_dispose() /opt/rh/gcc-toolset-11/root/usr/include/c++/11/bits/shared_ptr_base.h:528
    #1 0xdd17f6 in ats_malloc(unsigned long) ../src/tscore/ink_memory.cc:65
    #8 0x58e416 in std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release() /opt/rh/gcc-toolset-11/root/usr/include/c++/11/bits/shared_ptr_base.h:168
    #9 0x58d9af in std::__shared_count<(__gnu_cxx::_Lock_policy)2>::~__shared_count() /opt/rh/gcc-toolset-11/root/usr/include/c++/11/bits/shared_ptr_base.h:705
    #2 0x5229b9 in RamCacheCLFUS::_resize_hashtable() ../src/iocore/cache/RamCacheCLFUS.cc:192
    #10 0xdf1313 in std::__shared_ptr<ts::Metrics::Storage, (__gnu_cxx::_Lock_policy)2>::~__shared_ptr() /opt/rh/gcc-toolset-11/root/usr/include/c++/11/bits/shared_ptr_base.h:1154
    #3 0x522e32 in RamCacheCLFUS::init(long, StripeSM*) ../src/iocore/cache/RamCacheCLFUS.cc:223
    #11 0xdf132f in std::shared_ptr<ts::Metrics::Storage>::~shared_ptr() /opt/rh/gcc-toolset-11/root/usr/include/c++/11/bits/shared_ptr.h:122
    #12 0x7f26409b4d4b in __run_exit_handlers (/lib64/libc.so.6+0x50d4b)
    #4 0x46e045 in store_compress_get ../src/iocore/cache/unit_tests/test_RamCacheCLFUS.cc:157

    #5 0x46f559 in CATCH2_INTERNAL_TEST_0 ../src/iocore/cache/unit_tests/test_RamCacheCLFUS.cc:201
previously allocated by thread T0 here:
    #6 0x7f2641c19080 in invoke ../lib/Catch2/src/catch2/internal/catch_test_registry.cpp:60
    #7 0x7f2641be5741 in Catch::TestCaseHandle::invoke() const ../lib/Catch2/src/catch2/catch_test_case_info.hpp:124
    #0 0x7f2643971307 in operator new(unsigned long) (/lib64/libasan.so.6+0xb6307)
    #8 0x7f2641be227b in Catch::RunContext::invokeActiveTestCase() ../lib/Catch2/src/catch2/internal/catch_run_context.cpp:673
    #9 0x7f2641be19f0 in Catch::RunContext::runCurrentTest() ../lib/Catch2/src/catch2/internal/catch_run_context.cpp:631
    #1 0xdf2c19 in std::_MakeUniq<std::tuple<std::array<std::tuple<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, int>, 1024ul>, std::array<ts::Metrics::AtomicType, 1024ul> > >::__single_object std::make_unique<std::tuple<std::array<std::tuple<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, int>, 1024ul>, std::array<ts::Metrics::AtomicType, 1024ul> >>() /opt/rh/gcc-toolset-11/root/usr/include/c++/11/bits/unique_ptr.h:962
    #10 0x7f2641bdc948 in Catch::RunContext::runTest(Catch::TestCaseHandle const&) ../lib/Catch2/src/catch2/internal/catch_run_context.cpp:273
    #2 0xdf1a15 in ts::Metrics::Storage::Storage() ../include/tsutil/Metrics.h:319
    #11 0x7f2641b21a28 in execute ../lib/Catch2/src/catch2/catch_session.cpp:108
    #3 0xdfb5f8 in decltype (::new ((void*)(0)) ts::Metrics::Storage()) std::construct_at<ts::Metrics::Storage>(ts::Metrics::Storage*) /opt/rh/gcc-toolset-11/root/usr/include/c++/11/bits/stl_construct.h:97
    #12 0x7f2641b2472f in Catch::Session::runInternal() ../lib/Catch2/src/catch2/catch_session.cpp:328
    #4 0xdfb642 in void std::allocator_traits<std::allocator<ts::Metrics::Storage> >::construct<ts::Metrics::Storage>(std::allocator<ts::Metrics::Storage>&, ts::Metrics::Storage*) /opt/rh/gcc-toolset-11/root/usr/include/c++/11/bits/alloc_traits.h:518
    #13 0x7f2641b23ca4 in Catch::Session::run() ../lib/Catch2/src/catch2/catch_session.cpp:260
    #5 0xdfaadc in std::_Sp_counted_ptr_inplace<ts::Metrics::Storage, std::allocator<ts::Metrics::Storage>, (__gnu_cxx::_Lock_policy)2>::_Sp_counted_ptr_inplace<>(std::allocator<ts::Metrics::Storage>) /opt/rh/gcc-toolset-11/root/usr/include/c++/11/bits/shared_ptr_base.h:519
    #6 0xdf953e in std::__shared_count<(__gnu_cxx::_Lock_policy)2>::__shared_count<ts::Metrics::Storage, std::allocator<ts::Metrics::Storage>>(ts::Metrics::Storage*&, std::_Sp_alloc_shared_tag<std::allocator<ts::Metrics::Storage> >) /opt/rh/gcc-toolset-11/root/usr/include/c++/11/bits/shared_ptr_base.h:650
    #7 0xdf7bd9 in std::__shared_ptr<ts::Metrics::Storage, (__gnu_cxx::_Lock_policy)2>::__shared_ptr<std::allocator<ts::Metrics::Storage>>(std::_Sp_alloc_shared_tag<std::allocator<ts::Metrics::Storage> >) /opt/rh/gcc-toolset-11/root/usr/include/c++/11/bits/shared_ptr_base.h:1342
    #8 0xdf5fe5 in std::shared_ptr<ts::Metrics::Storage>::shared_ptr<std::allocator<ts::Metrics::Storage>>(std::_Sp_alloc_shared_tag<std::allocator<ts::Metrics::Storage> >) /opt/rh/gcc-toolset-11/root/usr/include/c++/11/bits/shared_ptr.h:409
    #9 0xdf4265 in std::shared_ptr<ts::Metrics::Storage> std::allocate_shared<ts::Metrics::Storage, std::allocator<ts::Metrics::Storage>>(std::allocator<ts::Metrics::Storage> const&) /opt/rh/gcc-toolset-11/root/usr/include/c++/11/bits/shared_ptr.h:863
    #10 0xdf2d6d in std::shared_ptr<ts::Metrics::Storage> std::make_shared<ts::Metrics::Storage>() /opt/rh/gcc-toolset-11/root/usr/include/c++/11/bits/shared_ptr.h:879
    #11 0xdedee3 in ts::Metrics::instance() ../src/tsutil/Metrics.cc:39
    #12 0xd2be8a in RecLookupRecord(char const*, void (*)(RecRecord const*, void*), void*, bool) ../src/records/RecCore.cc:520
    #14 0x7f26444c74e7 in int Catch::Session::run<char>(int, char const* const*) ../lib/Catch2/src/catch2/catch_session.hpp:49
    #13 0xd2ba97 in RecGetRecordStringAlloc[abi:cxx11](char const*, bool) ../src/records/RecCore.cc:477
    #15 0x7f26444c72ce in main ../lib/Catch2/src/catch2/internal/catch_main.cpp:36
    #16 0x7f264099e864 in __libc_start_main (/lib64/libc.so.6+0x3a864)

Direct leak of 4064 byte(s) in 4 object(s) allocated from:
    #0 0x7f264396f9a7 in __interceptor_malloc (/lib64/libasan.so.6+0xb49a7)
    #1 0xdd17f6 in ats_malloc(unsigned long) ../src/tscore/ink_memory.cc:65
    #2 0x5229b9 in RamCacheCLFUS::_resize_hashtable() ../src/iocore/cache/RamCacheCLFUS.cc:192
    #3 0x522e32 in RamCacheCLFUS::init(long, StripeSM*) ../src/iocore/cache/RamCacheCLFUS.cc:223
    #4 0x46e045 in store_compress_get ../src/iocore/cache/unit_tests/test_RamCacheCLFUS.cc:157
    #14 0x56ed81 in configure_net ../src/iocore/net/Net.cc:59
    #5 0x470c93 in CATCH2_INTERNAL_TEST_4 ../src/iocore/cache/unit_tests/test_RamCacheCLFUS.cc:229
    #15 0x570a57 in ink_net_init(ts::ModuleVersion) ../src/iocore/net/Net.cc:139
    #6 0x7f2641c19080 in invoke ../lib/Catch2/src/catch2/internal/catch_test_registry.cpp:60
    #7 0x7f2641be5741 in Catch::TestCaseHandle::invoke() const ../lib/Catch2/src/catch2/catch_test_case_info.hpp:124
    #16 0x451f33 in EventProcessorListener::testRunStarting(Catch::TestRunInfo const&) (/home/jenkins/workspace/Github_Builds/rocky/src/build/src/iocore/cache/RamCacheCLFUS+0x451f33)
    #8 0x7f2641be227b in Catch::RunContext::invokeActiveTestCase() ../lib/Catch2/src/catch2/internal/catch_run_context.cpp:673
    #17 0x7f2641ae7227 in Catch::MultiReporter::testRunStarting(Catch::TestRunInfo const&) ../lib/Catch2/src/catch2/reporters/catch_reporter_multi.cpp:89
    #9 0x7f2641be19f0 in Catch::RunContext::runCurrentTest() ../lib/Catch2/src/catch2/internal/catch_run_context.cpp:631
    #18 0x7f2641bdbccf in Catch::RunContext::RunContext(Catch::IConfig const*, Catch::Detail::unique_ptr<Catch::IEventListener>&&) ../lib/Catch2/src/catch2/internal/catch_run_context.cpp:207
    #10 0x7f2641bdc948 in Catch::RunContext::runTest(Catch::TestCaseHandle const&) ../lib/Catch2/src/catch2/internal/catch_run_context.cpp:273
    #11 0x7f2641b21a28 in execute ../lib/Catch2/src/catch2/catch_session.cpp:108
    #19 0x7f2641b20dce in TestGroup ../lib/Catch2/src/catch2/catch_session.cpp:79
    #12 0x7f2641b2472f in Catch::Session::runInternal() ../lib/Catch2/src/catch2/catch_session.cpp:328
    #20 0x7f2641b246af in Catch::Session::runInternal() ../lib/Catch2/src/catch2/catch_session.cpp:327
    #13 0x7f2641b23ca4 in Catch::Session::run() ../lib/Catch2/src/catch2/catch_session.cpp:260
    #21 0x7f2641b23ca4 in Catch::Session::run() ../lib/Catch2/src/catch2/catch_session.cpp:260
    #14 0x7f26444c74e7 in int Catch::Session::run<char>(int, char const* const*) ../lib/Catch2/src/catch2/catch_session.hpp:49
    #22 0x7f26444c74e7 in int Catch::Session::run<char>(int, char const* const*) ../lib/Catch2/src/catch2/catch_session.hpp:49
    #15 0x7f26444c72ce in main ../lib/Catch2/src/catch2/internal/catch_main.cpp:36
    #23 0x7f26444c72ce in main ../lib/Catch2/src/catch2/internal/catch_main.cpp:36
    #16 0x7f264099e864 in __libc_start_main (/lib64/libc.so.6+0x3a864)

    #24 0x7f264099e864 in __libc_start_main (/lib64/libc.so.6+0x3a864)

Direct leak of 1016 byte(s) in 1 object(s) allocated from:
    #0 0x7f264396f9a7 in __interceptor_malloc (/lib64/libasan.so.6+0xb49a7)
Thread T1 ([ET_NET 0]) created by T0 here:
    #1 0xdd17f6 in ats_malloc(unsigned long) ../src/tscore/ink_memory.cc:65
    #0 0x7f26439137c5 in pthread_create (/lib64/libasan.so.6+0x587c5)
    #2 0x5229b9 in RamCacheCLFUS::_resize_hashtable() ../src/iocore/cache/RamCacheCLFUS.cc:192
    #1 0xd00da3 in ink_thread_create ../include/tscore/ink_thread.h:129
    #3 0x522e32 in RamCacheCLFUS::init(long, StripeSM*) ../src/iocore/cache/RamCacheCLFUS.cc:223
    #2 0xd0142b in Thread::start(char const*, void*, unsigned long, std::function<void ()> const&) ../src/iocore/eventsystem/Thread.cc:92
    #4 0x46e045 in store_compress_get ../src/iocore/cache/unit_tests/test_RamCacheCLFUS.cc:157
    #3 0xd0d1c7 in EventProcessor::spawn_event_threads(int, int, unsigned long) ../src/iocore/eventsystem/UnixEventProcessor.cc:472
    #5 0x4719d4 in CATCH2_INTERNAL_TEST_8 ../src/iocore/cache/unit_tests/test_RamCacheCLFUS.cc:244
    #4 0xd0dd52 in EventProcessor::start(int, unsigned long) ../src/iocore/eventsystem/UnixEventProcessor.cc:553
    #6 0x7f2641c19080 in invoke ../lib/Catch2/src/catch2/internal/catch_test_registry.cpp:60
    #5 0x45202f in EventProcessorListener::testRunStarting(Catch::TestRunInfo const&) (/home/jenkins/workspace/Github_Builds/rocky/src/build/src/iocore/cache/RamCacheCLFUS+0x45202f)
    #7 0x7f2641be5741 in Catch::TestCaseHandle::invoke() const ../lib/Catch2/src/catch2/catch_test_case_info.hpp:124
    #6 0x7f2641ae7227 in Catch::MultiReporter::testRunStarting(Catch::TestRunInfo const&) ../lib/Catch2/src/catch2/reporters/catch_reporter_multi.cpp:89
    #8 0x7f2641be227b in Catch::RunContext::invokeActiveTestCase() ../lib/Catch2/src/catch2/internal/catch_run_context.cpp:673
    #7 0x7f2641bdbccf in Catch::RunContext::RunContext(Catch::IConfig const*, Catch::Detail::unique_ptr<Catch::IEventListener>&&) ../lib/Catch2/src/catch2/internal/catch_run_context.cpp:207
    #9 0x7f2641be19f0 in Catch::RunContext::runCurrentTest() ../lib/Catch2/src/catch2/internal/catch_run_context.cpp:631
    #8 0x7f2641b20dce in TestGroup ../lib/Catch2/src/catch2/catch_session.cpp:79
    #10 0x7f2641bdc948 in Catch::RunContext::runTest(Catch::TestCaseHandle const&) ../lib/Catch2/src/catch2/internal/catch_run_context.cpp:273
    #9 0x7f2641b246af in Catch::Session::runInternal() ../lib/Catch2/src/catch2/catch_session.cpp:327
    #11 0x7f2641b21a28 in execute ../lib/Catch2/src/catch2/catch_session.cpp:108
    #10 0x7f2641b23ca4 in Catch::Session::run() ../lib/Catch2/src/catch2/catch_session.cpp:260
    #12 0x7f2641b2472f in Catch::Session::runInternal() ../lib/Catch2/src/catch2/catch_session.cpp:328
    #11 0x7f26444c74e7 in int Catch::Session::run<char>(int, char const* const*) ../lib/Catch2/src/catch2/catch_session.hpp:49
    #13 0x7f2641b23ca4 in Catch::Session::run() ../lib/Catch2/src/catch2/catch_session.cpp:260
    #12 0x7f26444c72ce in main ../lib/Catch2/src/catch2/internal/catch_main.cpp:36
    #14 0x7f26444c74e7 in int Catch::Session::run<char>(int, char const* const*) ../lib/Catch2/src/catch2/catch_session.hpp:49
    #13 0x7f264099e864 in __libc_start_main (/lib64/libc.so.6+0x3a864)
    #15 0x7f26444c72ce in main ../lib/Catch2/src/catch2/internal/catch_main.cpp:36

    #16 0x7f264099e864 in __libc_start_main (/lib64/libc.so.6+0x3a864)

SUMMARY: AddressSanitizer: 10160 byte(s) leaked in 10 allocation(s).
SUMMARY: AddressSanitizer: heap-use-after-free /opt/rh/gcc-toolset-11/root/usr/include/c++/11/bits/atomic_base.h:636 in std::__atomic_base<long>::fetch_add(long, std::memory_order)

@phongn

phongn commented Jun 16, 2026

Copy link
Copy Markdown
Collaborator Author

Related PR to add zstd and lz4 to our CI hosts: apache/trafficserver-ci#441

@phongn
phongn force-pushed the lz4 branch 2 times, most recently from 5d3ea03 to aa2a6a8 Compare June 24, 2026 16:21

@cmcfarlen cmcfarlen left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this looks good. I have a couple of concerns.

  1. Will this build if LZ4 or ZSTD are not found?
  2. I don't think CLFUS is used due to other concerns with the technique. Does this PR resolve those?

requesting changes just to get answers to the questions.

@phongn

phongn commented Jul 8, 2026

Copy link
Copy Markdown
Collaborator Author

I think this looks good. I have a couple of concerns.

  1. Will this build if LZ4 or ZSTD are not found?
  2. I don't think CLFUS is used due to other concerns with the technique. Does this PR resolve those?

requesting changes just to get answers to the questions.

  1. This will build without them, we have CMake-gated preprocessor macros guarding them (also used by the existing LZMA suppot)
  2. This PR does not resolve CLFUS issues but I intend to refactor Cache so that compression may be used with any other algorithm (like S3-FIFO). I had a proposed PR Make the CLFUS RAM cache adapt to a shifting working set #13235 to try and fix CLFUS but it's such an intrusive change that I decided to try different algorithms instead.

@phongn
phongn requested a review from cmcfarlen July 8, 2026 20:45
cmcfarlen
cmcfarlen previously approved these changes Jul 13, 2026

@bryancall bryancall left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The threads from my June review are all resolved and CI is green, so the only thing blocking this now is the merge conflict.

Please rebase onto master and I will do the full review. With 22 files touched, including CMakeLists.txt, both Dockerfiles and the new FindLZ4.cmake / FindZSTD.cmake modules, I would rather review the rebased tree than a version that cannot land.

Marking this as request changes until it is mergeable again. That is not a comment on the code itself, which looked good in June.

phongn and others added 8 commits September 14, 2026 21:44
RamCacheCLFUS allocated its hash table, seen filter, and entries but had
no destructor, so destroying an instance leaked them. LeakSanitizer
flagged this once the new unit test started creating and destroying
instances. Add a destructor that releases each entry's data, returns the
entries to the allocator, and frees the table and seen filter.

The synchronous RAM cache test never calls TEST_DONE(), so the event
threads kept running as the process tore down static state at exit and
an ET_NET thread incremented the freed Metrics singleton
(heap-use-after-free). Shut the event system down from the shared cache
test harness at end of run so the threads stop first.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

@bryancall bryancall left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Request changes

Thanks for the rebase. The June threads are all addressed and the rebased tree looks good, but one thing outside the diff blocks the feature from working as shipped.

Blocking: the records range check still rejects 4 and 5

src/records/RecordsConfig.cc line 889 defines proxy.config.cache.ram_cache.compress with the validity pattern [0-3]. RecYAMLDecoder runs that check on load, logs a validity warning on failure, and uses the default of 0. So setting 4 or 5 in records.yaml, as the docs now tell people to, silently leaves compression off. The unit test does not catch it because it writes cache_config_ram_cache_compress directly. Changing the pattern to [0-5] is the fix. It would be worth a one-line autest or gold-test config that sets the record to 5 and checks traffic_layout or a startup log line, so the range check cannot drift again.

Fixed: sticky-null zstd context

The one-time warning in zstd_cctx() / zstd_dctx() covers what I asked for. One non-blocking refinement: when zstd_cctx() returns null in compress_entries(), failed = true lands on Lfailed, which sets incompressible on the entry. That records a thread-level allocation condition as a permanent property of the data, which is the opposite of the (correct) choice you made in get(). goto Lcontinue there would leave the entry eligible for a later pass. A ram_cache.compress.failure counter next to the new decompress one would also make this diagnosable without log access.

Fixed: decode failures are now visible

The throttled warning plus the global and per-volume ram_cache.decompress.failure counters, with the null-context case treated as a miss, is exactly right. Non-blocking: the codec's own error is dropped on the floor in every branch. Capturing ZSTD_getErrorName() or the negative lz4 return into a local and printing it in the Lfailed warning would tell an operator whether a nonzero counter means a corrupted frame or a bookkeeping bug in len.

Fixed: test proves compression happened

256 KB payload and size_after < size_before for every non-NONE backend. Two non-blocking gaps: the incompressible test does not assert size_after == size_before, so a regression that stored a raw copy instead of marking the entry would still pass, and the single-byte case runs only under CACHE_COMPRESSION_NONE, so lz4 and zstd on tiny input (both emit frames larger than 1 byte) are untested. Parametrizing it over compression_cases() like the others would close that. Also, nothing discriminates the liblzma lzma_stream_buffer_bound change; the old code failed the encode and the new code stores raw, and both read back as NONE. Fine to leave, but the description should call it a robustness fix rather than a memory fix.

Fixed: static_asserts

Two non-blocking notes. static_assert(CACHE_COMPRESSION_ZSTD < (1 << 3)) sits inside #ifdef HAVE_ZSTD_H in the .cc, so a build without zstd never evaluates it; it belongs in RamCacheCLFUS.h directly under the bitfield, unconditional. The six pairwise asserts in Cache.h are each true by construction and would not fire if someone added CACHE_COMPRESSION_FOO 6 without the matching enumerator; one static_assert(RAM_HIT_LAST_ENTRY == CACHE_COMPRESSION_ZSTD + 2) would.

Non-blocking: the new destructor and the scheduled compressor

~RamCacheCLFUS() frees the entries, buckets and seen filter, but init() schedules a RamCacheCLFUSCompressor holding a raw back-pointer that nothing cancels. Production never destroys these objects, and the test sidesteps it by initializing with compression off, so this is latent. Either keep the Event * and cancel it in the destructor, or say in a comment that the destructor is only safe when compression was never scheduled. The comment in test_RamCacheCompressEntries.cc saying the policy has no destructor is now stale.

Not verified by running it

The new test cannot be run at the base commit since it depends on symbols this patch adds, so there is no red-without-fix result to report. CI is green on the rebased head.

Allow the new codecs to actually be configured. The validity pattern for
proxy.config.cache.ram_cache.compress was still [0-3], so RecYAMLDecoder
rejected 4 and 5 at load time, logged a validity warning and fell back to
the default of 0 -- leaving compression off for exactly the two backends
the docs now recommend. Widen it to [0-5] and add a records unit test that
walks every CACHE_COMPRESSION_* value against the record's own check and
pattern, so the range cannot drift behind the enum again.

Distinguish a missing zstd context from incompressible data. A null
ZSTD_CCtx is a thread/allocator condition, so recording it as the entry's
permanent incompressible flag was the opposite of the choice already made
in get(). The entry is now left eligible for a later pass. Because that
failure is sticky and the compressor event is pinned to one ET_TASK
thread, leaving entries eligible alone would turn a one-time allocation
failure into a walk over most of the RAM cache every second -- dropping
and retaking the stripe lock and allocating a compressBound()-sized buffer
per entry, only to fail each time -- so compress_entries() now skips the
whole pass on a thread with no context. Real compression failures and
skipped passes increment a new ram_cache.compress.failure counter (global
and per-volume) so they are diagnosable without log access; objects that
merely did not shrink enough are deliberately not counted, since that is
the ordinary outcome for already-compressed content.

Report the codec's own error on a decompression failure. Every branch
dropped it, so a nonzero decompress.failure counter could not tell a
corrupt frame from a bookkeeping error in e->len. The throttled warning
now carries ZSTD_getErrorName(), zError(), or the codec's numeric return.

Move the bitfield static_assert next to the field it guards, in
RamCacheCLFUS.h and unconditional; inside #ifdef HAVE_ZSTD_H it was never
evaluated by a build without zstd. Add one count assert in Cache.h that
fires if a CACHE_COMPRESSION_* is added without its RAM_HIT_COMPRESS_*
enumerator; the pairwise asserts are kept because they catch a reordering
of either sequence, which the count assert does not.

Strengthen the compression tests: the incompressible case now asserts the
entry's footprint is unchanged (the 256 KB payload is a power of two, so
there is no padding for the pass to legitimately reclaim), and the
single-byte case is parametrized over every backend instead of running
only under CACHE_COMPRESSION_NONE.

Document why ~RamCacheCLFUS() is only safe for a cache that never
scheduled the background compressor, and correct the now-stale comment in
test_RamCacheCompressEntries.cc. Cancelling the event is not enough to fix
this: the continuation carries no mutex, so it can run compress_entries()
concurrently with the destructor. Making that safe means giving the
compressor the stripe mutex and requiring the destructor to hold it, which
is not worth it while production never destroys a RamCacheCLFUS.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
@phongn

phongn commented Sep 15, 2026

Copy link
Copy Markdown
Collaborator Author

Thanks — good catch on the range check, that one would have shipped a feature nobody could turn on. All six points are addressed in 421b518. Details, including two places where I went past what you asked and one where I deliberately didn't.

Blocking: records range check — fixed

[0-3][0-5]. Confirmed the mechanism end to end before changing it: RECC_INT goes to recordRangeCheck(), which parses the pattern as a numeric min-max (not a character class — [131072-104857600] elsewhere in the file makes that unambiguous), and RecYAMLDecoder.cc:167 on failure takes the ERRATA_WARN path and returns without ever calling RecSetRecord, so the record keeps its "0" default. Exactly as you described.

For the drift guard I used a records unit test rather than an autest:

TEST_CASE("ram_cache.compress accepts every compression backend", "[librecords][RecUtils]")
{
  const auto *record = GetRecordElementByName("proxy.config.cache.ram_cache.compress");
  ...
  for (int i = CACHE_COMPRESSION_NONE; i <= CACHE_COMPRESSION_ZSTD; i++) {
    INFO("CACHE_COMPRESSION_* value: " << i);
    REQUIRE(RecordValidityCheck(std::to_string(i).c_str(), record->check, record->regex));
  }
  REQUIRE_FALSE(RecordValidityCheck(std::to_string(CACHE_COMPRESSION_ZSTD + 1).c_str(), record->check, record->regex));
}

Two reasons over a gold test: it iterates the real CACHE_COMPRESSION_* values, so adding a codec without widening the pattern fails the build's own test rather than depending on someone remembering to extend a config; and it follows the existing search_default_domains accepts documented values test immediately above it, which does the same GetRecordElementByName + record->check/record->regex dance. Cost is one new include — iocore/cache/Cache.h into a records unit test. Those are header-only macros so it links clean in every config I tried, but if you consider that layering objectionable I'll hardcode 0..5 with a comment instead; it just loses the automatic-failure property.

This one I did verify red-without-fix: reverted the pattern to [0-3], rebuilt, and the test fails pointing at value 4.

Sticky-null zstd context — fixed, and the fix is bigger than the one you suggested

goto Lcontinue was right about intent, but taking it literally at the per-entry level would have traded a silent degradation for a worse operational problem, so I want to flag the reasoning rather than bury it.

init() schedules the compressor with eventProcessor.schedule_every(..., HRTIME_SECOND, ET_TASK). Periodic events are re-enqueued on the same EThread (UnixEThread.cc, the if (e->period) branch), so a given cache's compressor runs on exactly one ET_TASK thread for the process lifetime. The context failure is thread_local and sticky. So for a cache whose compressor landed on the unlucky thread, "leave the entry eligible" means: every second, walk up to compress_percent of the entries, and for each one drop the stripe lock, ats_malloc(ZSTD_compressBound(e->len)), fail, free, retake the lock. Forever. That is a lock-thrashing and allocator-churn regression against the current behaviour, which at least gives up quietly after one pass.

So compress_entries() now checks once at the top and skips the whole pass:

#ifdef HAVE_ZSTD_H
  if (cache_config_ram_cache_compress == CACHE_COMPRESSION_ZSTD && zstd_cctx() == nullptr) {
    ts::Metrics::Counter::increment(cache_rsb.ram_cache_compress_failures);
    ts::Metrics::Counter::increment(stripe->cache_vol->vol_rsb.ram_cache_compress_failures);
    return;
  }
#endif

Entries are untouched, so they stay eligible exactly as you wanted; one counter increment per skipped pass keeps it visible without flooding. The per-entry no_context handling is still there underneath as defence in depth, with a comment saying the pass-level check makes it unreachable for zstd today.

ram_cache.compress.failure added, global and per-volume, documented on both stats pages. It counts codec errors and context-allocation failures; it deliberately does not count objects that only failed required_shrink, since that is the normal outcome for already-compressed content and would drown the signal.

Decode failures — codec error now reported

All seven goto Lfailed sites in get() set a detail string first. zstd uses ZSTD_getErrorName(), libz uses zError(), and the rest format their numeric return plus what was expected, e.g. LZ4_decompress_safe returned %d, expected %d — which separates a malformed frame (negative) from e->len disagreeing with the frame (smaller non-negative), which was your actual question.

One wording change while I was in there: the default: arm said "compression type not supported by this build", but CacheProcessor::cacheInitialized() already Fatals on a compiled-out backend, so reaching that arm means a corrupt flag_bits.compressed, not a build-option problem. It now says "no decoder for this compression type".

Tests — strengthened, with one honest limitation

Incompressible case asserts size_after == size_before; single-byte case is now parametrized over compression_cases() like the others, so lz4 and zstd on 1-byte input are covered (they all converge on RAM_HIT_COMPRESS_NONE, whether via fastlz's len < 16 guard, the incompressible marking, or the verbatim-store path).

On the footprint assertion — it is weaker than it looks and I'd rather say so than let it read as more than it is. It catches "stored the expanded compressed blob", but it cannot distinguish "marked incompressible and left alone" from "re-stored a raw copy", because the 256 KB payload is a power of two and carries no buffer padding, so both land on the same size(). With a padded payload the raw re-store would legitimately shrink the entry, so there's no assertion that covers both. The comment in the test now states this rather than claiming it catches the raw-copy case.

You're also right that nothing discriminates the liblzma change, and for a stronger reason than the test being weak: lzma_easy_buffer_encode() is handed the output size and returns LZMA_BUF_ERROR rather than overrunning, so the old code was never unsafe, and old-fails-encode and new-stores-raw are indistinguishable at the get() boundary. I've rewritten that bullet in the description to call it a robustness fix and to spell out that it is neither a memory-safety nor a memory-saving change.

static_asserts — fixed, with one deviation

Bitfield assert moved to RamCacheCLFUS.h directly under the field, unconditional. Verified it is now actually evaluated without zstd: in a -DCMAKE_DISABLE_FIND_PACKAGE_ZSTD=ON build, temporarily falsifying it does fail the compile, which it could not do from inside #ifdef HAVE_ZSTD_H.

Added your count assert. I did not remove the six pairwise ones, though: they catch a reordering of either sequence (reorder the RAM_HIT_COMPRESS_* enumerators and the count assert still passes while the pairwise ones fire), whereas yours catches a CACHE_COMPRESSION_* added without its enumerator. Different failures, both cheap, so I kept both and wrote a comment saying which does what. Happy to drop the pairwise block if you'd still rather have just the one.

Destructor and the scheduled compressor — took the comment option

I went with documenting the constraint rather than cancelling, because cancelling isn't sufficient and I didn't want to ship something that looks safe and isn't. RamCacheCLFUSCompressor is constructed with no mutex, so Event::cancel() from the destructor's thread races a compress_entries() already executing on the ET_TASK thread — you'd close the "fires again later" window and leave the "currently running against a half-destroyed object" one open. Making it genuinely safe means giving the compressor stripe->mutex and requiring the destructor to hold it, which changes production scheduling behaviour (the compressor would then contend and reschedule on lock miss) for a path production never takes — these live as long as their StripeSM. ~RamCacheCLFUS() now carries that reasoning, and the stale comment in test_RamCacheCompressEntries.cc is corrected. Also wired both failure counters in that test, which had been left with null metric pointers.

If you'd rather have the mutex change, I'd prefer it as its own PR against the shared-compression refactor, where the compressor continuation moves anyway.

Verification

Three configurations, all green: default, ENABLE_ASAN (clean — the June leak and use-after-free stay fixed), and -DCMAKE_DISABLE_FIND_PACKAGE_{LZ4,ZSTD,LibLZMA}=ON to exercise the compiled-out paths. Full-tree build is warning-clean.

Worth noting for the record: this host has lz4 1.9.3 and zstd 1.5.5, so all six cases actually ran (79 assertions, versus the 45 you'd have seen in the Rocky CI log where lz4 was missing and the test WARNed about it). apache/trafficserver-ci#441 is still the thing that closes that gap in CI.

Deliberately not in this PR

The libz and liblzma decode paths check only the library return code, never that the output length equals e->len — fastlz, lz4 and zstd all do. Pre-existing and outside this diff, and the right place to fix it once for every codec is the shared decompress() helper in the follow-up refactor, so I've queued it there rather than adding a sixth thing here.

clang-analyzer flagged the "no detail" initializer for codec_error as a
dead store, correctly: every path to Lfailed assigned a real detail string
first, so the sentinel was never read. Dropping just the initializer would
have left a genuine uninitialized read the first time someone added a
goto Lfailed without setting it.

The sentinel and the function-scope char buffer only existed because the
detail string had to survive a goto, so report the failure at each site
instead. note_decompress_failure() now carries the throttled warning and
both counters, each branch formats its detail in its own scope, and
Lfailed is back to freeing the buffer and destroying the entry. Behavior
is unchanged: one process-wide throttler, and the entry is still intact
when the warning reads its fields.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@phongn

phongn commented Sep 15, 2026

Copy link
Copy Markdown
Collaborator Author

Follow-up in aaf29f1 for the two CI failures on 421b518.

Clang-Analyzer — fixed

deadcode.DeadStores on the codec_error sentinel I added for the decode-detail reporting:

RamCacheCLFUS.cc:308:15: warning: Value stored to 'codec_error' during its initialization is never read

Correct, and the sentinel was load-bearing for nothing: all seven paths to Lfailed assign a real detail string first, so "no detail" was never read. I didn't just delete the initializer — that trades a harmless dead store for a real uninitialized read the first time someone adds a goto Lfailed without assigning.

Instead I removed the reason the sentinel existed. The const char * and the function-scope char[128] were only there because the detail had to survive a goto, so each site now reports before jumping, with its buffer scoped to the branch:

if (l != rc) {
  char detail[128];
  snprintf(detail, sizeof(detail), "LZ4_decompress_safe returned %d, expected %d", rc, l);
  note_decompress_failure(stripe, key, e, detail);
  goto Lfailed;
}

A new file-local note_decompress_failure() holds the throttled Warning and both counters, and Lfailed is back to what it looked like before this PR touched it — ats_free(b); this->_destroy(e); plus the debug trace. Behaviour is unchanged: static Throttler inside the helper is still one process-wide throttler, and the helper is called while the entry is intact, so the warning reads the same e-> fields it read from inside the old Lfailed block. Net is 7 fewer lines and two fewer function-scope locals.

This supersedes the codec_error snippet in my previous comment — same information reaches the log, different plumbing.

Verified locally with clang-tidy --checks='-*,clang-analyzer-*' against a compile_commands.json (reproduced the warning first, confirmed it gone after), plus the .clang-tidy-ci check set clean on the file, and the cache tests green in all three configurations including ASan.

AuTest 2of4 — not this PR

The only failing test in that shard is rate_limit_sni_queue, on the precondition assertion:

file .../ts/log/traffic.out : a connection was queued - Failed
   Reason: ... did not contains expression: "Queueing the VC"

That is #13679, opened today: the queue precondition races a TLS handshake on a 0.3s timer. The shard was also heavily loaded on this run (worker 1 took ~1456s), which is the condition that issue describes. Nothing in this PR reaches the SNI rate limiter — the diff is RAM cache compression, a records validity pattern, and two counters — and every other shard passed, as did all nine platform builds.

I'd rather not rebase just to reroll the dice on a known flake, so unless you want it rerun for a clean board, the new push should sort it on its own.

phongn and others added 2 commits September 15, 2026 21:03
None of these change RAM cache behavior. The zstd config-mode alias
block could never fire, the decompression warning reimplemented the
site-wide log throttling it should have called, and the
compress.failure doc claimed a per-entry count that the pass-level
skip does not produce. The rpm spec and the contrib images also
lacked lz4 and zstd, so a documented compress value would be Fatal
at startup on those builds.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Several were mine and hid real behavior: compress.failure counted
skipped passes as entry failures, a transient codec allocation error
marked an entry permanently incompressible, the destructor skipped the
byte accounting _destroy() does, and the new destructor left the copy
operations implicit. Codec validation now happens once where the value
is read instead of twice after the compressor is already scheduled, and
the Find modules are the tree's own rather than a BSD import.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@phongn

phongn commented Sep 16, 2026

Copy link
Copy Markdown
Collaborator Author

Correcting something I told you in my earlier reply, since the claim is wrong and it is load-bearing for a decision you may make later.

On the destructor thread, I wrote:

Making it genuinely safe means giving the compressor stripe->mutex and requiring the destructor to hold it, which changes production scheduling behaviour (the compressor would then contend and reschedule on lock miss) for a path production never takes

The stripe mutex is exactly the wrong mutex for this, and not for the reason I gave. Mutex_unlock() only decrements nthread_holding and releases the underlying lock when that reaches zero (include/iocore/eventsystem/Lock.h:363-368). compress_entries() relies on actually dropping the stripe lock around the codec call: it takes the lock on entry, MUTEX_UNTAKE_LOCKs before compressing, and retakes it afterwards. If the continuation were dispatched already holding stripe->mutex, that inner untake would take the count from 2 to 1 rather than releasing, so every codec call would run with the stripe locked. That is a throughput regression on the RAM cache hit path, which is considerably worse than the contend-and-reschedule cost I described.

The right shape is the one used elsewhere in the tree: give RamCacheCLFUSCompressor its own new_ProxyMutex(), keep the Event * that schedule_every() returns, and cancel it under that mutex in the destructor. QUICPacketHandler's _collector_event and PreWarmManager's _tick_event both do this. That also removes the reason the destructor currently only documents its precondition: with a retained Event * it can enforce it.

Worth being explicit about why the cancel is needed at all, which I was vague about: EventProcessor::schedule only copies a mutex onto the event if the continuation has one (UnixEventProcessor.cc:704-706), and the lock helpers treat a null mutex as already acquired (Lock.h:518-519). So RamCacheCLFUSCompressor, having no mutex, is dispatched with no synchronisation against anything at all today.

I have not made that change here — it is a behavioural change to production scheduling and belongs with the shared-compression refactor, where the compressor continuation moves anyway. I have queued it there, and corrected the comment in ~RamCacheCLFUS() that repeated the same bad advice. My conclusion for this PR is unchanged: document the precondition rather than ship a cancel that looks safe and is not.

bryancall
bryancall previously approved these changes Sep 16, 2026

@bryancall bryancall left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ Approving. The blocker from my last review is fixed and it now has a test that catches it.

I rebuilt and ran this rather than reading it: configured 132ddc21 on macOS/clang with lz4 1.10.0 and zstd 1.5.7, ran RamCacheCLFUS (79 assertions, all six backends) and test_records "[RecUtils]", then reconfigured with both CMAKE_DISABLE_FIND_PACKAGE_* set and reran. Everything I call verified below was run.

I reverted src/records/RecordsConfig.cc:889 to [0-3], rebuilt, and got test_RecUtils.cc:235: FAILED ... CACHE_COMPRESSION_* value: 4. Restored: 89 assertions green. The upper bound is 5 and CACHE_COMPRESSION_ZSTD is 5, so there is no off-by-one hiding zstd.

All of my earlier asks have landed. Two worth calling out as better than what I asked for: the null-cctx handling became a pass-level skip at RamCacheCLFUS.cc:549 plus a transient flag routing allocation errors to Lcontinue, and the error-text capture covers all five codecs rather than the new two. On the incompressible case you used size_after <= size_before instead of == with a comment about padding: I ran it and it is exactly equal at 262280 both sides, your reasoning is right, leave it.

Also verified, since these are the things most likely to be wrong in a change like this: the zstd contexts are thread_local std::unique_ptr with ZSTD_freeCCtx/ZSTD_freeDCtx deleters so they release at thread exit rather than leaking, a per-call ZSTD_CCtx_reset() is genuinely unnecessary because ZSTD_compress2() starts a fresh frame and the level is sticky, the read path treats every codec failure as a miss with the entry intact and no double free, and buffer sizing uses LZ4_compressBound() / ZSTD_compressBound() with the codec's own return stored rather than the bound.

Non-blocking

1. Deleting the zstd config-mode shim is a regression against master

a8a1c45a removed the alias block because it "could never fire". It does fire.

With CMAKE_FIND_PACKAGE_PREFER_CONFIG=ON, find_package(ZSTD 1.4.0) at CMakeLists.txt:496 resolves through zstd's own config package instead of cmake/FindZSTD.cmake. ZSTD_FOUND is set so HAVE_ZSTD_H is true, but the exported target is zstd::libzstd_shared, not zstd::zstd. cmake/FindZSTD.cmake:70 never runs, nothing creates the alias, and the four places linking zstd::zstd fail at generate time (src/iocore/cache/CMakeLists.txt:66, src/iocore/net/CMakeLists.txt:137, src/traffic_layout/CMakeLists.txt:35, plugins/compress/CMakeLists.txt:29).

Reproduced standalone against this tree's cmake/ directory:

-- PREFER_CONFIG ZSTD_FOUND=1 ZSTD_CONFIG=/opt/homebrew/lib/cmake/zstd/zstdConfig.cmake
-- TARGET zstd::libzstd_shared EXISTS
-- *** zstd::zstd MISSING
CMake Error at CMakeLists.txt:10 (target_link_libraries):
    zstd::zstd

It is a non-default setting and no ATS CI job sets it, which is why CI is green, so I am not blocking. I would still fix it before merge: it is four lines after find_package(ZSTD 1.4.0), aliasing whichever of zstd::libzstd_shared / zstd::libzstd_static / zstd::libzstd exists.

The description still says the alias shim is kept for CMAKE_FIND_PACKAGE_PREFER_CONFIG builds. Please either restore it or drop that sentence, and fix the description before re-requesting review, since the next reviewer reads it alongside the diff.

2. Nothing tests the decompress failure paths

note_decompress_failure() at RamCacheCLFUS.cc:295 and its five call sites are the visible part of this change on the read path, and no test reaches any of them. I proved it: replacing the ZSTD_isError(ll) check at line 398 and the l != rc check at line 377 with if (false) still gives "All tests passed (79 assertions in 4 test cases)". Deleting both codecs' error detection is invisible to the suite.

A corrupted-frame case closes it: store, run compress_entries(), flip a byte in the stored blob, get(), then assert a miss, an intact entry, and ram_cache_decompress_failures == 1. The entries are private so it needs a small test hook or a friend declaration. Worth it, because the counter and the warning are the operator-facing half of this feature and right now they are only proven to compile.

3. A real round-trip failure will report as a stringification crash

CHECK(r.out == payload) at test_RamCacheCLFUS.cc:206, 238 and 265 compares two 256 KB std::vector<char>. Catch2 stringifies both operands to report an assertion and on this payload that throws. ./RamCacheCLFUS -s gives due to unexpected exception with messages: compression backend: zstd / basic_string.

Green without -s, so not a failing test today. The problem is the day a codec genuinely breaks the round trip and CI shows unexpected exception: basic_string instead of which backend and which byte. Compare into a bool first, or assert on sizes plus memcmp.

4. The lz4 version floor is below the API the tree uses

CMakeLists.txt:501 sets 1.7.0, but src/traffic_layout/info.cc:273 calls LZ4_versionString(), which lz4.h documents as requiring v1.7.5+. Against 1.7.0 through 1.7.4 the find module reports success and traffic_layout then fails to link. Raise the floor to 1.7.5, or print only LZ4_VERSION_STRING. The zstd 1.4.0 floor is correctly enforced: find_package(ZSTD 99.0.0) gives Could NOT find ZSTD: Found unsuitable version "1.5.7".

5. The Fatal message does not say what to change

Cache.cc:884 and 889 say "lz4 not available for RAM cache compression". The operator hitting this just set a number in records.yaml and now has a process that will not start. Name the record and the value the way the default: case at line 893 does.

6. Smaller things

  • NOTICE gains one trailing blank line and no content. Drop it from the diff.
  • ci/docker/yum/Dockerfile renames zstd-devel to libzstd-devel on a line unrelated to lz4. Probably the right Fedora name, but it is an unrelated change riding along.
  • doc/developer-guide/cache-architecture/ram-cache.en.rst fixes uncompressible to incompressible and leaves "it mat be compressed" on the next line.
  • traffic_layout info --versions prints zstd as the runtime version and lz4 as the compile-time one with a separate lz4.run. Harmless, but similar names now mean different things.

One thing to answer before this merges

@cmcfarlen's dismissed review asked two questions. The first, whether this builds without lz4 or zstd, I have verified myself: it does, cleanly, and the test skips with a WARN rather than failing. The second, whether this addresses the other concerns with the CLFUS technique, has no answer anywhere in the thread or the description. He is a requested reviewer again, so please answer it in the PR rather than leaving my approval to carry the change past it.

On CI

All 14 checks pass on 132ddc21. One caveat worth stating: the unit test WARNs rather than failing when a backend is absent, so a green run does not by itself prove the new code was exercised anywhere in CI, and your own description notes the Fedora image still needs lz4-devel. That is why I built and ran it locally.

Restoring the config-mode alias is a fix for my own regression: with
CMAKE_FIND_PACKAGE_PREFER_CONFIG the lookup resolves through zstd's
config package on a case-insensitive filesystem, which exports
zstd::libzstd_shared rather than zstd::zstd, so deleting the alias
broke macOS builds at generate time. The new corrupted-frame test
reaches the decompression failure paths, which nothing exercised
before, and the round-trip checks no longer hand Catch2 two 256 KB
operands to stringify, which threw instead of reporting the mismatch.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@phongn

phongn commented Sep 16, 2026

Copy link
Copy Markdown
Collaborator Author

Thanks — and thank you for rebuilding it rather than reading it. All six are addressed in ebb773b8. Taking them in order of how wrong I was.

1. The zstd shim — you are right, and it was my regression

I deleted it on a Linux-only check and generalised. CMake's config-mode search looks for ZSTDConfig.cmake or zstd-config.cmake; zstd installs zstdConfig.cmake. On a case-sensitive filesystem that matches neither, which is what I tested, and I concluded "dead everywhere" — missing that ZSTDConfig.cmake and zstdConfig.cmake are the same file on a case-insensitive one. Your Homebrew repro is the case I reasoned away.

I reproduced it here rather than taking it on faith, by simulating the config package on Linux and toggling only the filename:

config filename present ZSTD_FOUND zstd::zstd
zstdConfig.cmake only (case-sensitive) FALSE
also matching ZSTDConfig.cmake (what macOS gives you) 1 MISSING

So HAVE_ZSTD_H goes true and the four link sites fail at generate time, exactly as you described.

Restored as a loop over zstd::libzstd_shared / _static / libzstd. One change from the original: if the config package exports none of those names, it now warns and clears HAVE_ZSTD_H instead of leaving a dangling zstd::zstd, so the failure mode is "built without zstd" rather than a generate error. Verified both paths against the simulated package: zstd::zstd RESOLVES with HAVE_ZSTD_H=1 under PREFER_CONFIG, and the module path unchanged.

Description fixed too, and expanded — the sentence you flagged was true again once the shim came back, but it did not say why the shim exists, which is how it got deleted. It now spells out the case-insensitive filename match.

2. Decompress failure paths — test added, and it earned its place

You were right that nothing reached them. CLFUS reports a corrupted compressed entry rather than serving it stores a payload, runs compress_entries(), overwrites the stored blob, then asserts a miss, ram_cache_decompress_failures incremented by exactly one, and the entry gone from the cache. Parametrized over all five codecs. Reaching the entry needs a friend struct RamCacheCLFUSTestAccess on RamCacheCLFUS, declared with a comment saying nothing in the product uses it.

It found more than coverage: it exercises the per-codec detail text for real, and all five produce a distinct, useful message.

uncompress: data error
lzma_stream_buffer_decode returned 7, wrote 0 of 262144 output bytes
LZ4_decompress_safe returned -1050, expected 262144
ZSTD_decompressDCtx: Unknown frame descriptor

I re-ran your mutation. if (false) on the lz4 l != rc check now fails the test. Your zstd mutation still passes, and there is a reason worth recording: that path has two checks, and disabling ZSTD_isError(ll) alone leaves l != ll to catch it — the corrupt frame is still never served, it just reports "produced 18446744073709551606 bytes, expected 262144" instead of the error name. Disabling both does fail the test. So zstd is covered; your single-check mutation survived because of the second check, not a gap.

3. The 256 KB compare — worse than green-without--s

./RamCacheCLFUS -s was not merely unhelpful, it was a hard failure: test_RamCacheCLFUS.cc:238: FAILED: due to unexpected exception with messages: basic_string::_M_create, with a screenful of raw payload bytes before it. -s passes cleanly now.

Replaced with a size check plus a first_difference() helper, so a real round-trip break reports the offset as a number — 65 == 262144 rather than an exception.

4. lz4 floor — raised to 1.7.5

Raised. I could not verify the requirement from anything on this host: lz4's NEWS never mentions versionString and the 1.9.3 header carries no since annotation, so I am taking 1.7.5 from you and your 1.10.0 header. It costs nothing on any supported distro, so I would rather have it wrong-and-conservative than be right by accident. find_package(LZ4 1.7.5) reports found suitable version "1.9.3", minimum required is "1.7.5", and raising it to 99.0.0 correctly refuses.

5. Fatal message — now names the record and the value

proxy.config.cache.ram_cache.compress is 5 (zstd), but this build has no zstd support
proxy.config.cache.ram_cache.compress has unknown value 7

6. Smaller things

  • NOTICE is byte-identical to master again. The stray newline was mine, from removing the VTK attribution when the modules stopped being derived work.
  • ci/docker/yum/Dockerfile reverted to zstd-devel; only the lz4-devel addition remains. Worth flagging separately though, because I think it is a real bug rather than just an unrelated change: there is no zstd-devel on EL9 and libzstd-devel does not provide it, so that line looks like it cannot install as written. It came in with Add Zstandard compression support and update tests #12201, predates this PR, and belongs in its own change.
  • it mat be compressedmay.
  • traffic_layout --versions now prints lz4 as the runtime version and drops lz4.run, so lz4 and zstd mean the same thing. I deliberately did not change what zstd reports, since that key has meant runtime since Add Zstandard compression support and update tests #12201 and someone may be parsing it.

cmcfarlen's second question

It was answered, just not anywhere you would find it: this comment from 2026-07-08, about fourteen months up the thread and before the rebase. Short version is that this PR does not resolve the CLFUS concerns, and the intent is to refactor so compression is available to any RAM cache algorithm. I will leave it to @phongn whether that wants restating for @cmcfarlen directly.

On CI coverage

Agreed, and it is the reason I keep reporting local numbers. This host has lz4 1.9.3 and zstd 1.5.5, so all six backends run: 131 assertions now, up from 79, and 59 in a CMAKE_DISABLE_FIND_PACKAGE_* build. ASan clean, both full-tree builds clean, clang-analyzer-* and .clang-tidy-ci clean on every file I touched. apache/trafficserver-ci#441 is still what closes the gap for the Fedora image.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants