Regression tests fail with -R 3

[bneradt]

Our CI jobs currently run the regression tests with -R 1 and the tests are pretty stable with that option. However, we have noticed that they fail due to a , at least in CI, if we pass -R 3. We should get to the bottom of why it's failing with -R 3 and configure at least some of the regression tests with -R 3 since that's a superset of -R 1.

Reproduction Steps

docker run -it -u 1200:1200 --name regression_failure_repro ci.trafficserver.apache.org/ats/rockylinux:8 /bin/bash

# Now, from within that container.

cd /var/tmp
git clone https://github.com/apache/trafficserver.git
cd trafficserver/
source /opt/rh/gcc-toolset-11/enable
autoreconf -fiv
./configure --with-openssl=/opt/openssl-quic --prefix=/tmp/ats/ --enable-werror --enable-debug
make -j `nproc`
make install
/tmp/ats/bin/traffic_server -K -k -R 3

Expected Results

The tests should run and pass

What Happens

It fails with a crash:

REGRESSION TEST SDK_API_HttpParentProxySet_Fail started
Regression test(SDK_API_HttpParentProxySet_Fail) still in progress
RPRINT SDK_API_HttpParentProxySet_Fail: setting synserver parent proxy to 127.0.0.1:3300
Fatal: HttpTransact.h:611: failed assertion `_v <= configured_connect_attempts_max_retries`
[Mar 31 21:39:22.254] traffic_crashlo NOTE: crashlog started, target=34868, debug=false syslog=true, uid=1200 euid=1200
[Mar 31 21:39:22.254] traffic_crashlo NOTE: logging to 0x13170c0
[Mar 31 21:39:22.318] traffic_crashlo ERROR: wrote crash log to /tmp/ats/var/log/trafficserver/crash-2023-03-31-213922.log
traffic_server: received signal 6 (Aborted)
traffic_server - STACK TRACE: 
/tmp/ats/bin/traffic_server(_Z19crash_logger_invokeiP9siginfo_tPv+0xc3)[0x95bc55]
/lib64/libpthread.so.0(+0x12ce0)[0x7f8ad7acace0]
/lib64/libc.so.6(gsignal+0x10f)[0x7f8ad6c0ea4f]
/lib64/libc.so.6(abort+0x127)[0x7f8ad6be1db5]
/tmp/ats//lib/libtscore.so.10(_Z11ink_warningPKcz+0x0)[0x7f8ad9150423]
/tmp/ats//lib/libtscore.so.10(_Z17ats_base64_encodePKhmPcmPm+0x0)[0x7f8ad914b01e]
/tmp/ats/bin/traffic_server(_ZN12HttpTransact12_CurrentInfo8Attempts9incrementEl+0x4a)[0xab1154]
/tmp/ats/bin/traffic_server(_ZN12HttpTransact27handle_response_from_parentEPNS_5StateE+0xc86)[0xa8ea52]
/tmp/ats/bin/traffic_server(_ZN12HttpTransact14HandleResponseEPNS_5StateE+0xb5b)[0xa8d551]
/tmp/ats/bin/traffic_server(_ZN6HttpSM32call_transact_and_set_next_stateEPFvPN12HttpTransact5StateEE+0x78)[0xa4fa86]
/tmp/ats/bin/traffic_server(_ZN6HttpSM17handle_api_returnEv+0x177)[0xa2ab87]
/tmp/ats/bin/traffic_server(_ZN6HttpSM17state_api_calloutEiPv+0xe70)[0xa2a82a]
/tmp/ats/bin/traffic_server(_ZN6HttpSM18state_api_callbackEiPv+0x289)[0xa298fd]
/tmp/ats/bin/traffic_server(TSHttpTxnReenable+0x109)[0x992957]
/tmp/ats/bin/traffic_server[0x9e22d1]
/tmp/ats/bin/traffic_server(_ZN15INKContInternal12handle_eventEiPv+0x236)[0x984b22]
/tmp/ats/bin/traffic_server(_ZN12Continuation11handleEventEiPv+0xdd)[0x962053]
/tmp/ats/bin/traffic_server(_ZNK7APIHook6invokeEiPv+0xeb)[0x98579d]
/tmp/ats/bin/traffic_server(_ZN6HttpSM17state_api_calloutEiPv+0xa0b)[0xa2a3c5]
/tmp/ats/bin/traffic_server(_ZN6HttpSM23do_api_callout_internalEv+0x21d)[0xa4515d]
/tmp/ats/bin/traffic_server(_ZN6HttpSM14do_api_calloutEv+0x27)[0xa59d23]
/tmp/ats/bin/traffic_server(_ZN6HttpSM33state_read_server_response_headerEiPv+0xd2a)[0xa2dca0]
/tmp/ats/bin/traffic_server(_ZN6HttpSM12main_handlerEiPv+0x36b)[0xa32c09]
/tmp/ats/bin/traffic_server(_ZN12Continuation11handleEventEiPv+0xdd)[0x962053]
/tmp/ats/bin/traffic_server[0xd84370]
/tmp/ats/bin/traffic_server[0xd84d4e]
/tmp/ats/bin/traffic_server[0xd85407]
/tmp/ats/bin/traffic_server(_ZN18UnixNetVConnection11net_read_ioEP10NetHandlerP7EThread+0x2b)[0xd87b0f]
/tmp/ats/bin/traffic_server(_ZN10NetHandler18process_ready_listEv+0x9b)[0xd781c7]
/tmp/ats/bin/traffic_server(_ZN10NetHandler15waitForActivityEl+0x4e0)[0xd788da]
/tmp/ats/bin/traffic_server(_ZN7EThread15execute_regularEv+0x462)[0xedd578]
/tmp/ats/bin/traffic_server(_ZN7EThread7executeEv+0x107)[0xedd71f]
/tmp/ats/bin/traffic_server[0xedc002]
/lib64/libpthread.so.0(+0x81cf)[0x7f8ad7ac01cf]
/lib64/libc.so.6(clone+0x43)[0x7f8ad6bf9d83]
Aborted

[bneradt]

Once this is fixed, we should configure -R 3 to at least some CI regression test runs.

[ezelkow1]

Also the expensive tests are broken, from a test run on rocky with them enabled:

/opt/rh/gcc-toolset-11/root/usr/bin/ld: /opt/rh/gcc-toolset-11/root/usr/lib/gcc/x86_64-redhat-linux/11/libstdc++_nonshared.a(sstream-inst80.o): undefined reference to symbol '_ZTVNSt7__cxx1119basic_ostringstreamIwSt11char_traitsIwESaIwEEE@@GLIBCXX_3.4.21'
/opt/rh/gcc-toolset-11/root/usr/bin/ld: /usr/lib64/libstdc++.so.6: error adding symbols: DSO missing from command line```


with `--enable-expensive-tests`

[lzx404243]

The crashing test is SDK_API_HttpParentProxySet_Fail. The crash can be reproduced running this test alone:

./build/bin/traffic_server -K -k -R 3 -r SDK_API_HttpParentProxySet_Fail

Looks like the test was passing until #9366 merged in. Will look into root causing this.

[lzx404243]

I have a fix in mind to prevent this. Will put up a PR.