ATS9: Crash on ProxySession::do_api_callout()

[masaori335]

[ 0  ] libpthread-2.17.so     waitpid                                                      ( :undefined               ) 
[ 1  ] traffic_server         crash_logger_invoke(int, siginfo_t*, void*)                  ( Crash.cc:168             ) 
[ 2  ] libpthread-2.17.so     0x7fc6cfa2f5f0                                               ( :undefined               ) 
[ 3  ]                       0x0                                                          ( :undefined               ) 
[ 4  ] traffic_server         ProxySession::do_api_callout(TSHttpHookID)                   ( ProxySession.cc:171      ) 
[ 5  ] traffic_server         Http1ClientSession::destroy()                                ( Http1ClientSession.cc:79 ) 
[ 6  ] traffic_server         HttpSM::kill_this()                                          ( HttpSM.cc:6946           ) 
[ 7  ] traffic_server         HttpSM::state_api_callback(int, void*)                       ( HttpSM.cc:1351           ) 
[ 8  ] traffic_server         TSHttpTxnReenable                                            ( InkAPI.cc:6096           ) 
[ 9  ] authproxy.so (deleted) 0x7fc692d66878                                               ( :undefined               ) 
[ 10 ] authproxy.so (deleted) 0x7fc692d65d15                                               ( :undefined               ) 
[ 11 ] traffic_server         INKContInternal::handle_event(int, void*)                    ( InkAPI.cc:1101           ) 
[ 12 ] traffic_server         PluginVC::process_read_side(bool)                            ( I_Continuation.h:190     ) 
[ 13 ] traffic_server         PluginVC::process_write_side(bool)                           ( PluginVC.cc:569          ) 
[ 14 ] traffic_server         PluginVC::main_handler(int, void*)                           ( PluginVC.cc:225          ) 
[ 15 ] traffic_server         EThread::process_event(Event*, int)                          ( I_Continuation.h:190     ) 
[ 16 ] traffic_server         EThread::process_queue(Queue<Event, Event::Link_link>*, i... ( UnixEThread.cc:176       ) 
[ 17 ] traffic_server         EThread::execute_regular()                                   ( UnixEThread.cc:236       ) 
[ 18 ] traffic_server         EThread::execute()                                           ( UnixEThread.cc:345       ) 
[ 19 ] traffic_server         spawn_thread_internal(void*)                                 ( Thread.cc:92             ) 
[ 20 ] libpthread-2.17.so     start_thread                                                 ( :undefined               )

[masaori335]

Some details

(gdb) bt full
#0  0x00007fc6cfa2f199 in waitpid () from /lib64/libpthread.so.0
No symbol table info available.
#1  0x000055dc92e3d394 in crash_logger_invoke (signo=11, info=0x7fc6c690acb0, ctx=0x7fc6c690ab80) at traffic_server/Crash.cc:168
        status = 0
#2  <signal handler called>
No symbol table info available.
#3  0x0000000000000000 in ?? ()
No symbol table info available.
#4  0x000055dc92e8b912 in Http1ClientSession::free (this=0x7fc6178e50c0) at Http1ClientSession.cc:115
No locals.
#5  0x000055dc930372de in handle_api_return (event=60000, this=<optimized out>) at ProxySession.cc:171
        hookid = TS_HTTP_SSN_CLOSE_HOOK
#6  ProxySession::do_api_callout (this=0x7fc6178e50c0, id=<optimized out>) at ProxySession.cc:148
No locals.
#7  0x000055dc92e8b839 in Http1ClientSession::destroy (this=0x7fc6178e50c0) at Http1ClientSession.cc:79
No locals.
#8  0x000055dc92ea27d9 in HttpSM::kill_this (this=0x7fc60b0aa3a0) at HttpSM.cc:6946
No locals.
#9  0x000055dc92ea8dc9 in HttpSM::state_api_callback (this=0x7fc60b0aa3a0, event=60000, data=0x7fc63053b820) at HttpSM.cc:1351
No locals.
#10 0x000055dc92e56d11 in TSHttpTxnReenable (txnp=0x7fc60b0aa3a0, event=TS_EVENT_HTTP_CONTINUE) at traffic_server/InkAPI.cc:6096
        trylock = {m = {m_ptr = 0x7fc62de160f0}, lock_acquired = true}
        sm = 0x7fc60b0aa3a0
        eth = 0x7fc6ca60d000
        cb = <optimized out>
#11 0x00007fc692d66878 in ?? ()
No symbol table info available.
#12 0x0000000000000001 in ?? ()
No symbol table info available.
#13 0x00007fc5fab8f508 in ?? ()
No symbol table info available.
#14 0x00007fc6c690bb50 in ?? ()
No symbol table info available.
#15 0x00007fc692d65d15 in ?? ()
No symbol table info available.
#16 0xfffffffffffffff8 in ?? ()
No symbol table info available.
#17 0x00007fc6a0b50300 in ?? ()
No symbol table info available.
#18 0x00007fc5fab8f508 in ?? ()
No symbol table info available.
#19 0x00007fc69dbffa00 in ?? ()
No symbol table info available.
#20 0x00007fc6c690bbb0 in ?? ()
No symbol table info available.
#21 0x000055dc92e490d8 in INKContInternal::handle_event (this=0x7fc63a8373e0, event=100, edata=0x7fc6305e7420) at traffic_server/InkAPI.cc:1101
        retval = <optimized out>

(gdb) frame 4
#4  0x000055dc92e8b912 in Http1ClientSession::free (this=0x7fc6178e50c0) at Http1ClientSession.cc:115
115	  this->do_io_write(nullptr, 0, nullptr);
(gdb) p client_vc
$5 = (NetVConnection *) 0x7fc62f184ea0

[shinrich]

Sounds like the crashes we were seeing that should have been addressed by #6469. But I see that commit is already in the 9.0.x branch.

[zwoop]

Mark T. and I looked over our builds, and everything prior to this should be in the build that we see this crasher with:

commit 919b9fa6b3d7366cda181aab0172c32308945360
Author:     Brian Neradt <brian.neradt@gmail.com>
AuthorDate: Tue Mar 10 21:31:26 2020 -0500
Commit:     Leif Hedstrom <zwoop@apache.org>
CommitDate: Tue Mar 10 20:31:41 2020 -0600

This includes the fixes in #6469 ...

[masaori335]

Duplicated with #6527. Closing this because #6527 has more details (symbols, history...)