8.1.x crash with collapsed forwarding enabled (HandleCacheOpenReadHitFreshness)

[ezelkow1]

I am sure this crash was due to having collapsed forwarding enabled (proxy.config.http.cache.open_write_fail_action = 5), its exactly like some crashes we had seen when I was testing my homegrown version of this, and was never able to find the issue. In our own version I had commented out this assert, no idea what the real issue is though since I could never track it down

[ 0 ] libpthread-2.17.so __libc_waitpid ( :undefined )
[ 1 ] traffic_server crash_logger_invoke ( Crash.cc:165 )
[ 2 ] libpthread-2.17.so 0x2ab80bd4c5e0 ( :undefined )
[ 3 ] libc-2.17.so raise ( :undefined )
[ 4 ] libc-2.17.so __GI_abort ( :undefined )
[ 5 ] libtscore.so.8.1.0 ink_abort ( ink_error.cc:99 )
[ 6 ] libtscore.so.8.1.0 _ink_assert ( ink_assert.cc:37 )
[ 7 ] traffic_server HttpTransact::HandleCacheOpenReadHitFreshness(HttpTransac... ( HttpTransact.cc:2320 )
[ 8 ] traffic_server HttpSM::call_transact_and_set_next_state(void ()(HttpTra... ( HttpSM.cc:7156 )
[ 9 ] traffic_server HttpSM::handle_api_return() ( HttpSM.cc:1590 )
[ 10 ] traffic_server HttpSM::state_api_callout(int, void) ( HttpSM.cc:1525 )
[ 11 ] traffic_server HttpSM::set_next_state() ( HttpSM.cc:7198 )
[ 12 ] traffic_server HttpSM::call_transact_and_set_next_state(void ()(HttpTra... ( HttpSM.cc:7164 )
[ 13 ] traffic_server HttpSM::state_cache_open_read(int, void) ( HttpSM.cc:2499 )
[ 14 ] traffic_server HttpSM::main_handler(int, void*) ( HttpSM.cc:2561 )
[ 15 ] traffic_server handleEvent ( I_Continuation.h:160 )
[ 16 ] traffic_server HttpCacheSM::state_cache_open_read(int, void*) ( HttpCacheSM.cc:130 )
[ 17 ] traffic_server handleEvent ( I_Continuation.h:160 )
[ 18 ] traffic_server callcont ( P_CacheInternal.h:643 )
[ 19 ] traffic_server CacheVC::openReadStartEarliest(int, Event*) ( CacheRead.cc:999 )
[ 20 ] traffic_server handleEvent ( I_Continuation.h:160 )
[ 21 ] traffic_server CacheVC::openReadStartEarliest(int, Event*) ( CacheRead.cc:993 )
[ 22 ] traffic_server CacheVC::openReadStartHead(int, Event*) ( CacheRead.cc:1260 )
[ 23 ] traffic_server handleEvent ( I_Continuation.h:160 )
[ 24 ] traffic_server CacheVC::handleReadDone(int, Event*) ( Cache.cc:2397 )
[ 25 ] traffic_server handleEvent ( I_Continuation.h:160 )
[ 26 ] traffic_server Cache::open_read(Continuation*, ats::CryptoHash const*, H... ( CacheRead.cc:162 )
[ 27 ] traffic_server CacheProcessor::open_read(Continuation*, HttpCacheKey con... ( Cache.cc:3326 )
[ 28 ] traffic_server do_cache_open_read ( HttpCacheSM.cc:281 )
[ 29 ] traffic_server HttpCacheSM::open_read(HttpCacheKey const*, URL*, HTTPHdr... ( HttpCacheSM.cc:314 )
[ 30 ] traffic_server HttpSM::do_cache_lookup_and_read() ( HttpSM.cc:4509 )
[ 31 ] traffic_server HttpSM::set_next_state() ( HttpSM.cc:7334 )
[ 32 ] traffic_server HttpSM::call_transact_and_set_next_state(void ()(HttpTra... ( HttpSM.cc:7164 )
[ 33 ] traffic_server HttpSM::state_hostdb_reverse_lookup(int, void) ( HttpSM.cc:2257 )
[ 34 ] traffic_server HttpSM::main_handler(int, void*) ( HttpSM.cc:2561 )
[ 35 ] traffic_server handleEvent ( I_Continuation.h:160 )
[ 36 ] traffic_server reply_to_cont ( HostDB.cc:492 )
[ 37 ] traffic_server HostDBContinuation::dnsEvent(int, HostEnt*) ( HostDB.cc:1502 )
[ 38 ] traffic_server handleEvent ( I_Continuation.h:160 )
[ 39 ] traffic_server DNSEntry::postEvent(int, Event*) ( DNS.cc:1413 )
[ 40 ] traffic_server dns_result ( DNS.cc:1363 )
[ 41 ] traffic_server dns_process ( DNS.cc:1760 )
[ 42 ] traffic_server DNSHandler::recv_dns(int, Event*) ( DNS.cc:903 )
[ 43 ] traffic_server DNSHandler::mainEvent(int, Event*) ( DNS.cc:916 )
[ 44 ] traffic_server NetHandler::waitForActivity(long) ( UnixNet.cc:519 )
[ 45 ] traffic_server EThread::execute_regular() ( UnixEThread.cc:278 )
[ 46 ] traffic_server spawn_thread_internal ( Thread.cc:85 )
[ 47 ] libpthread-2.17.so start_thread ( :undefined )

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. Marking it stale to flag it for further consideration by the community.

[bryancall]

@ezelkow1 Is this still an issue? Should we close this?

[ezelkow1]

I suppose we can close this. The issue does still exist and will continue to be a problem on at least 8.1.x. We have not tried collapsed forwarding with any of the 9x releases yet to know if the behavior has changed at all

Up to you if we want to close it or not, since it is a known problem on a current release, but it doesnt seem like a pressing want by the community at this point

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. Marking it stale to flag it for further consideration by the community.

[bryancall]

This collapsed-forwarding crash was fixed in 2022 by #8613. The assert at the top of HandleCacheOpenReadHitFreshness fired because the open_write_fail_action=5 path rewound the state machine to re-run the cache lookup without resetting request_sent_time/response_received_time; #8613 adds that reset (present in current master), and the fail-action paths were hardened further in 2026 (#12845, #12852). 8.1.x is end-of-life; closing as fixed. Please retest on a supported release and reopen if you can still reproduce it.

Update: this fix first shipped in the 9.2.0 release; it is in the current 10.1.2 release.