[C++] Linux Flight/FlightSQL test randomly segfaults after gRPC version bump

[alinaliBQ]

Describe the bug, including details regarding any error messages, version, and platform.

After the following PR was merged:

• GH-48926: [C++] Upgrade Abseil/Protobuf/GRPC/Google-Cloud-CPP bundled versions #48964

Non-deterministic crash at Flight/Flight SQL/ODBC tests is introduced:

SIGSEGV in /lib64/ld-linux-x86-64.so.2

The crash can happen at arrow-flight-test, arrow-flight-sql-test, or arrow-odbc-spi-impl-test. They all use gRPC and protobuf. The common denominator is arrow_flight_testing_static or arrow_flight_testing_shared that all three tests are linked against.

arrow-flight-test failure: https://github.com/apache/arrow/actions/runs/24410900735/job/71307679982#step:6:10272
arrow-flight-sql-test failure: https://github.com/apache/arrow/actions/runs/24410900735/job/71318452958#step:6:10207
arrow-odbc-spi-impl-test failure: https://github.com/apache/arrow/actions/runs/24410900735/job/71307679982#step:6:10330
arrow-flight-sql-odbc-test failure in local repo: (logs might expire)
https://github.com/Bit-Quill/arrow/actions/runs/24807100725/job/72603729735#step:8:15251

The error log looks like:

 Test #68: arrow-flight-test .....................***Failed 0.54 sec
Running arrow-flight-test, redirecting output into /build/cpp/build/test-logs/arrow-flight-test.txt (attempt 1/1)
/arrow/cpp/build-support/run-test.sh: line 88: 26750 Segmentation fault (core dumped) $TEST_EXECUTABLE "$@" > $LOGFILE.raw 2>&1
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
Running '/build/cpp/release/arrow-flight-test' produced core dump at '/tmp/core.arrow-flight-te.26750', printing backtrace:
[New LWP 26750]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Core was generated by `/build/cpp/release/arrow-flight-test'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0 0x00007f643d998128 in ?? () from /lib64/ld-linux-x86-64.so.2

Thread 1 (Thread 0x7f643c215700 (LWP 26750)):
#0 0x00007f643d998128 in ?? () from /lib64/ld-linux-x86-64.so.2
#1 0x00007f643d9aa789 in ?? () from /lib64/ld-linux-x86-64.so.2
#2 0x00007f643d9a6f46 in ?? () from /lib64/ld-linux-x86-64.so.2
#3 0x00007f643d9a875e in ?? () from /lib64/ld-linux-x86-64.so.2
#4 0x00007f643d9a7548 in ?? () from /lib64/ld-linux-x86-64.so.2
#5 0x0000000000000001 in ?? ()
#6 0x00007fffa0cad7f7 in ?? ()
#7 0x0000000000000000 in ?? ()
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
/build/cpp/src/arrow/flight

Logs from arrow-flight-sql-odbc-test:
Also fails at /lib64/ld-linux-x86-64.so.2, so I think it is the same root cause

 Test #71: arrow-flight-sql-odbc-test .......***Failed 0.64 sec
Running arrow-flight-sql-odbc-test, redirecting output into /arrow/build/cpp/build/test-logs/arrow-flight-sql-odbc-test.txt (attempt 1/1)
/arrow/cpp/build-support/run-test.sh: line 88: 16539 Segmentation fault (core dumped) $TEST_EXECUTABLE "$@" > $LOGFILE.raw 2>&1
Running main() from gmock_main.cc
[==========] Running 701 tests from 47 test suites.
[----------] Global test environment set-up.
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
Running '/arrow/build/cpp/release/arrow-flight-sql-odbc-test' produced core dump at '/tmp/core.arrow-flight-sq.16539', printing backtrace:
[New LWP 16539]
[New LWP 16540]
[New LWP 16542]
[New LWP 16541]
[New LWP 16543]
[New LWP 16544]
[New LWP 16545]
[New LWP 16546]
[New LWP 16547]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Core was generated by `/arrow/build/cpp/release/arrow-flight-sql-odbc-test'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0 0x00007fa4afb16128 in ?? () from /lib64/ld-linux-x86-64.so.2
[Current thread is 1 (Thread 0x7fa4ae19a880 (LWP 16539))]

Thread 9 (Thread 0x7fa499ffd6c0 (LWP 16547)):
#0 0x00007fa4af348072 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
#1 0x000055c1b8d24696 in pollset_work(grpc_pollset*, grpc_pollset_worker**, grpc_core::Timestamp) ()
#2 0x000055c1b8d29797 in pollset_work(grpc_pollset*, grpc_pollset_worker**, grpc_core::Timestamp) ()
#3 0x000055c1b8d31047 in grpc_pollset_work(grpc_pollset*, grpc_pollset_worker**, grpc_core::Timestamp) ()
#4 0x000055c1b8d9ef3d in (anonymous namespace)::maybe_event_engine_pollset_work(grpc_pollset*, grpc_pollset_worker**, grpc_core::Timestamp) ()
#5 0x000055c1b8da071c in cq_next(grpc_completion_queue*, gpr_timespec, void*) ()
#6 0x000055c1b8935a2c in grpc::CompletionQueue::AsyncNextInternal(void**, bool*, gpr_timespec) ()
#7 0x000055c1b895d23d in grpc::Server::SyncRequestThreadManager::PollForWork(void**, bool*) ()
#8 0x000055c1b896aba0 in grpc::ThreadManager::MainWorkLoop() [clone .localalias] ()
#9 0x000055c1b896ae00 in grpc::ThreadManager::WorkerThread::WorkerThread(grpc::ThreadManager*)::{lambda(void*)#1}::_FUN(void*) ()
#10 0x000055c1b9050c73 in grpc_core::(anonymous namespace)::ThreadInternalsPosix::ThreadInternalsPosix(char const*, void (*)(void*), void*, bool*, grpc_core::Thread::Options const&)::{lambda(void*)#1}::_FUN(void*) ()
#11 0x00007fa4af2baaa4 in ?? () from /lib/x86_64-linux-gnu/libc.so.6
#12 0x00007fa4af347c6c in ?? () from /lib/x86_64-linux-gnu/libc.so.6

Thread 8 (Thread 0x7fa49affe6c0 (LWP 16546)):
#0 0x00007fa4af34528d in syscall () from /lib/x86_64-linux-gnu/libc.so.6
#1 0x000055c1b960d83a in absl::lts_20250127::synchronization_internal::FutexWaiter::WaitUntil(std::atomic<int>*, int, absl::lts_20250127::synchronization_internal::KernelTimeout) [clone .localalias] ()
#2 0x000055c1b960d8b3 in absl::lts_20250127::synchronization_internal::FutexWaiter::Wait(absl::lts_20250127::synchronization_internal::KernelTimeout) ()
#3 0x000055c1b960dae2 in AbslInternalPerThreadSemWait_lts_20250127 ()
#4 0x000055c1b96108da in absl::lts_20250127::CondVar::WaitCommon(absl::lts_20250127::Mutex*, absl::lts_20250127::synchronization_internal::KernelTimeout) [clone .cold] ()
#5 0x000055c1b8d18b11 in grpc_event_engine::experimental::WorkStealingThreadPool::WorkStealingThreadPoolImpl::Lifeguard::LifeguardMain() [clone .localalias] ()
#6 0x000055c1b9050cd0 in grpc_core::(anonymous namespace)::ThreadInternalsPosix::ThreadInternalsPosix(char const*, void (*)(void*), void*, bool*, grpc_core::Thread::Options const&)::{lambda(void*)#1}::_FUN(void*) ()
#7 0x00007fa4af2baaa4 in ?? () from /lib/x86_64-linux-gnu/libc.so.6
#8 0x00007fa4af347c6c in ?? () from /lib/x86_64-linux-gnu/libc.so.6

Thread 7 (Thread 0x7fa49bfff6c0 (LWP 16545)):
#0 0x00007fa4af34528d in syscall () from /lib/x86_64-linux-gnu/libc.so.6
#1 0x000055c1b960d83a in absl::lts_20250127::synchronization_internal::FutexWaiter::WaitUntil(std::atomic<int>*, int, absl::lts_20250127::synchronization_internal::KernelTimeout) [clone .localalias] ()
#2 0x000055c1b960d8b3 in absl::lts_20250127::synchronization_internal::FutexWaiter::Wait(absl::lts_20250127::synchronization_internal::KernelTimeout) ()
#3 0x000055c1b960dae2 in AbslInternalPerThreadSemWait_lts_20250127 ()
#4 0x000055c1b96108da in absl::lts_20250127::CondVar::WaitCommon(absl::lts_20250127::Mutex*, absl::lts_20250127::synchronization_internal::KernelTimeout) [clone .cold] ()
#5 0x000055c1b8d17d7e in grpc_event_engine::experimental::WorkStealingThreadPool::WorkSignal::WaitWithTimeout(grpc_core::Duration) [clone .localalias] ()
#6 0x000055c1b8d17f1c in grpc_event_engine::experimental::WorkStealingThreadPool::ThreadState::Step() [clone .localalias] ()
#7 0x000055c1b8d180d8 in grpc_event_engine::experimental::WorkStealingThreadPool::ThreadState::ThreadBody() [clone .localalias] ()
#8 0x000055c1b8d184c2 in grpc_event_engine::experimental::WorkStealingThreadPool::WorkStealingThreadPoolImpl::StartThread()::{lambda(void*)#1}::_FUN(void*) ()
#9 0x000055c1b9050cd0 in grpc_core::(anonymous namespace)::ThreadInternalsPosix::ThreadInternalsPosix(char const*, void (*)(void*), void*, bool*, grpc_core::Thread::Options const&)::{lambda(void*)#1}::_FUN(void*) ()
#10 0x00007fa4af2baaa4 in ?? () from /lib/x86_64-linux-gnu/libc.so.6
#11 0x00007fa4af347c6c in ?? () from /lib/x86_64-linux-gnu/libc.so.6

Thread 6 (Thread 0x7fa4a99bb6c0 (LWP 16544)):
#0 0x00007fa4af34528d in syscall () from /lib/x86_64-linux-gnu/libc.so.6
#1 0x000055c1b960d83a in absl::lts_20250127::synchronization_internal::FutexWaiter::WaitUntil(std::atomic<int>*, int, absl::lts_20250127::synchronization_internal::KernelTimeout) [clone .localalias] ()
#2 0x000055c1b960d8b3 in absl::lts_20250127::synchronization_internal::FutexWaiter::Wait(absl::lts_20250127::synchronization_internal::KernelTimeout) ()
#3 0x000055c1b960dae2 in AbslInternalPerThreadSemWait_lts_20250127 ()
#4 0x000055c1b96108da in absl::lts_20250127::CondVar::WaitCommon(absl::lts_20250127::Mutex*, absl::lts_20250127::synchronization_internal::KernelTimeout) [clone .cold] ()
#5 0x000055c1b8d17d7e in grpc_event_engine::experimental::WorkStealingThreadPool::WorkSignal::WaitWithTimeout(grpc_core::Duration) [clone .localalias] ()
#6 0x000055c1b8d17f1c in grpc_event_engine::experimental::WorkStealingThreadPool::ThreadState::Step() [clone .localalias] ()
#7 0x000055c1b8d180d8 in grpc_event_engine::experimental::WorkStealingThreadPool::ThreadState::ThreadBody() [clone .localalias] ()
#8 0x000055c1b8d184c2 in grpc_event_engine::experimental::WorkStealingThreadPool::WorkStealingThreadPoolImpl::StartThread()::{lambda(void*)#1}::_FUN(void*) ()
#9 0x000055c1b9050cd0 in grpc_core::(anonymous namespace)::ThreadInternalsPosix::ThreadInternalsPosix(char const*, void (*)(void*), void*, bool*, grpc_core::Thread::Options const&)::{lambda(void*)#1}::_FUN(void*) ()
#10 0x00007fa4af2baaa4 in ?? () from /lib/x86_64-linux-gnu/libc.so.6
Errors while running CTest
#11 0x00007fa4af347c6c in ?? () from /lib/x86_64-linux-gnu/libc.so.6

Thread 5 (Thread 0x7fa4aa9bc6c0 (LWP 16543)):
#0 0x00007fa4af34528d in syscall () from /lib/x86_64-linux-gnu/libc.so.6
#1 0x000055c1b960d83a in absl::lts_20250127::synchronization_internal::FutexWaiter::WaitUntil(std::atomic<int>*, int, absl::lts_20250127::synchronization_internal::KernelTimeout) [clone .localalias] ()
#2 0x000055c1b960d8b3 in absl::lts_20250127::synchronization_internal::FutexWaiter::Wait(absl::lts_20250127::synchronization_internal::KernelTimeout) ()
#3 0x000055c1b960dae2 in AbslInternalPerThreadSemWait_lts_20250127 ()
#4 0x000055c1b96108da in absl::lts_20250127::CondVar::WaitCommon(absl::lts_20250127::Mutex*, absl::lts_20250127::synchronization_internal::KernelTimeout) [clone .cold] ()
#5 0x000055c1b8d0cf9e in grpc_event_engine::experimental::TimerManager::WaitUntil(grpc_core::Timestamp) [clone .localalias] ()
#6 0x000055c1b8d0d1ff in void absl::lts_20250127::internal_any_invocable::RemoteInvoker<false, void, grpc_event_engine::experimental::TimerManager::MainLoop()::{lambda()#1}&>(absl::lts_20250127::internal_any_invocable::TypeErasedState*) ()
#7 0x000055c1b8d1a3f5 in grpc_event_engine::experimental::SelfDeletingClosure::Run() ()
#8 0x000055c1b8d17e4f in grpc_event_engine::experimental::WorkStealingThreadPool::ThreadState::Step() [clone .localalias] ()
#9 0x000055c1b8d180d8 in grpc_event_engine::experimental::WorkStealingThreadPool::ThreadState::ThreadBody() [clone .localalias] ()
#10 0x000055c1b8d184c2 in grpc_event_engine::experimental::WorkStealingThreadPool::WorkStealingThreadPoolImpl::StartThread()::{lambda(void*)#1}::_FUN(void*) ()
#11 0x000055c1b9050cd0 in grpc_core::(anonymous namespace)::ThreadInternalsPosix::ThreadInternalsPosix(char const*, void (*)(void*), void*, bool*, grpc_core::Thread::Options const&)::{lambda(void*)#1}::_FUN(void*) ()
#12 0x00007fa4af2baaa4 in ?? () from /lib/x86_64-linux-gnu/libc.so.6
#13 0x00007fa4af347c6c in ?? () from /lib/x86_64-linux-gnu/libc.so.6

Thread 4 (Thread 0x7fa4ac9be6c0 (LWP 16541)):
#0 0x00007fa4af34528d in syscall () from /lib/x86_64-linux-gnu/libc.so.6
#1 0x000055c1b960d83a in absl::lts_20250127::synchronization_internal::FutexWaiter::WaitUntil(std::atomic<int>*, int, absl::lts_20250127::synchronization_internal::KernelTimeout) [clone .localalias] ()
#2 0x000055c1b960d8b3 in absl::lts_20250127::synchronization_internal::FutexWaiter::Wait(absl::lts_20250127::synchronization_internal::KernelTimeout) ()
#3 0x000055c1b960dae2 in AbslInternalPerThreadSemWait_lts_20250127 ()
#4 0x000055c1b96108da in absl::lts_20250127::CondVar::WaitCommon(absl::lts_20250127::Mutex*, absl::lts_20250127::synchronization_internal::KernelTimeout) [clone .cold] ()
#5 0x000055c1b9052b1c in gpr_cv_wait ()
#6 0x000055c1b8d52bfa in timer_thread(void*) ()
#7 0x000055c1b9050c73 in grpc_core::(anonymous namespace)::ThreadInternalsPosix::ThreadInternalsPosix(char const*, void (*)(void*), void*, bool*, grpc_core::Thread::Options const&)::{lambda(void*)#1}::_FUN(void*) ()
#8 0x00007fa4af2baaa4 in ?? () from /lib/x86_64-linux-gnu/libc.so.6
#9 0x00007fa4af347c6c in ?? () from /lib/x86_64-linux-gnu/libc.so.6

Thread 3 (Thread 0x7fa4ab9bd6c0 (LWP 16542)):
#0 0x00007fa4af348072 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
#1 0x000055c1b8cdcfb9 in grpc_event_engine::experimental::Epoll1Poller::DoEpollWait(std::chrono::duration<long, std::ratio<1l, 1000000000l> >) [clone .localalias] ()
#2 0x000055c1b8cdd295 in grpc_event_engine::experimental::Epoll1Poller::Work(std::chrono::duration<long, std::ratio<1l, 1000000000l> >, absl::lts_20250127::FunctionRef<void ()>) [clone .localalias] ()
#3 0x000055c1b8cf11de in grpc_event_engine::experimental::PosixEventEngine::PollingCycle::PollerWorkInternal() [clone .localalias] ()
#4 0x000055c1b8d1a3f5 in grpc_event_engine::experimental::SelfDeletingClosure::Run() ()
#5 0x000055c1b8d18020 in grpc_event_engine::experimental::WorkStealingThreadPool::ThreadState::Step() [clone .localalias] ()
#6 0x000055c1b8d180d8 in grpc_event_engine::experimental::WorkStealingThreadPool::ThreadState::ThreadBody() [clone .localalias] ()
#7 0x000055c1b8d184c2 in grpc_event_engine::experimental::WorkStealingThreadPool::WorkStealingThreadPoolImpl::StartThread()::{lambda(void*)#1}::_FUN(void*) ()
#8 0x000055c1b9050cd0 in grpc_core::(anonymous namespace)::ThreadInternalsPosix::ThreadInternalsPosix(char const*, void (*)(void*), void*, bool*, grpc_core::Thread::Options const&)::{lambda(void*)#1}::_FUN(void*) ()
#9 0x00007fa4af2baaa4 in ?? () from /lib/x86_64-linux-gnu/libc.so.6
#10 0x00007fa4af347c6c in ?? () from /lib/x86_64-linux-gnu/libc.so.6

Thread 2 (Thread 0x7fa4adbff6c0 (LWP 16540)):
#0 0x00007fa4af2b6d71 in ?? () from /lib/x86_64-linux-gnu/libc.so.6
#1 0x00007fa4af2b97ed in pthread_cond_wait () from /lib/x86_64-linux-gnu/libc.so.6
#2 0x000055c1b906e22c in background_thread_sleep (tsdn=<optimized out>, interval=<optimized out>, info=<optimized out>) at src/background_thread.c:137
#3 background_work_sleep_once (ind=0, info=<optimized out>, tsdn=<optimized out>) at src/background_thread.c:229
#4 background_thread0_work (tsd=<optimized out>) at src/background_thread.c:374
#5 background_work (ind=<optimized out>, tsd=<optimized out>) at src/background_thread.c:412
#6 background_thread_entry (ind_arg=<optimized out>) at src/background_thread.c:444
#7 0x00007fa4af2baaa4 in ?? () from /lib/x86_64-linux-gnu/libc.so.6
#8 0x00007fa4af347c6c in ?? () from /lib/x86_64-linux-gnu/libc.so.6

Thread 1 (Thread 0x7fa4ae19a880 (LWP 16539)):
#0 0x00007fa4afb16128 in ?? () from /lib64/ld-linux-x86-64.so.2
#1 0x00007fa4afb13b16 in ?? () from /lib64/ld-linux-x86-64.so.2
#2 0x00007fa4afb0751c in _dl_catch_exception () from /lib64/ld-linux-x86-64.so.2
#3 0x00007fa4afb12d20 in ?? () from /lib64/ld-linux-x86-64.so.2
#4 0x00007fa4afb0751c in _dl_catch_exception () from /lib64/ld-linux-x86-64.so.2
#5 0x00007fa4afb13164 in ?? () from /lib64/ld-linux-x86-64.so.2
#6 0x00007fa4af2b61a4 in ?? () from /lib/x86_64-linux-gnu/libc.so.6
#7 0x00007fa4afb0751c in _dl_catch_exception () from /lib64/ld-linux-x86-64.so.2
#8 0x00007fa4afb07669 in ?? () from /lib64/ld-linux-x86-64.so.2
#9 0x00007fa4af2b5c83 in ?? () from /lib/x86_64-linux-gnu/libc.so.6
#10 0x00007fa4af2b625f in dlopen () from /lib/x86_64-linux-gnu/libc.so.6
#11 0x00007fa4af216c62 in ?? () from /lib/x86_64-linux-gnu/libltdl.so.7
#12 0x00007fa4af216148 in ?? () from /lib/x86_64-linux-gnu/libltdl.so.7
#13 0x00007fa4af218550 in ?? () from /lib/x86_64-linux-gnu/libltdl.so.7
#14 0x00007fa4af218ce9 in lt_dlopenadvise () from /lib/x86_64-linux-gnu/libltdl.so.7
#15 0x00007fa4afa7acf0 in ?? () from /lib/x86_64-linux-gnu/libodbc.so.2
#16 0x00007fa4afa7c5ef in ?? () from /lib/x86_64-linux-gnu/libodbc.so.2
#17 0x00007fa4afaa7652 in SQLDriverConnectW () from /lib/x86_64-linux-gnu/libodbc.so.2
#18 0x000055c1b7bfd9d8 in arrow::flight::sql::odbc::ODBCTestBase::ConnectWithString(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, void*&) ()
#19 0x000055c1b7bffb2a in arrow::flight::sql::odbc::ODBCTestBase::Connect(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, void*&, void*&, int) ()
#20 0x000055c1b7c06dad in arrow::flight::sql::odbc::OdbcTestEnvironment::SetUp() ()
#21 0x00007fa4af8ec2cb in testing::internal::UnitTestImpl::RunAllTests() () from /usr/local/lib/libarrow_gtest.so.1.17.0
#22 0x00007fa4af8fc017 in bool testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) () from /usr/local/lib/libarrow_gtest.so.1.17.0
#23 0x00007fa4af8e1bb8 in testing::UnitTest::Run() () from /usr/local/lib/libarrow_gtest.so.1.17.0
#24 0x00007fa4af903962 in main () from /usr/local/lib/libarrow_gmock_main.so.1.17.0
#25 0x00007fa4af2481ca in ?? () from /lib/x86_64-linux-gnu/libc.so.6
#26 0x00007fa4af24828b in __libc_start_main () from /lib/x86_64-linux-gnu/libc.so.6
#27 0x000055c1b7bf8025 in _start ()
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
/arrow/build/cpp/src/arrow/flight/sql/odbc/tests

Component(s)

FlightRPC, C++

[alinaliBQ]

cc @raulcd @lidavidm @kou
If you could have a look at this, it would be great. Thanks

[kou]

Can we use debug build not release build in CI for better backtrace?

[raulcd]

@alinaliBQ apart from @kou 's suggestion so we can better diagnose, have you been able to reproduce locally?

[alinaliBQ]

@kou I have a debug workflow run here: https://github.com/apache/arrow/actions/runs/24532172035/job/71718034555?pr=49777#step:6:8685. Here is a draft PR that adds a debug workflow: #49777)
The backtrace seems similar as release. I think it's because the loading failure occurs before the test cases run.

@raulcd I could not reproduce this issue locally. I have only seen it on CI with cpp_test.sh.

[raulcd]

I've been able to reproduce pretty consistently locally (5 failures out of 6) with:

UBUNTU=24.04 archery docker run -e ARROW_BUILD_TYPE=DEBUG ubuntu-cpp-odbc

I've also reproduced with the underlying docker command:

UBUNTU=24.04 docker compose run --rm -e ARROW_BUILD_TYPE=DEBUG ubuntu-cpp-odbc

But when I try to run an interactive shell in the container with:

$ UBUNTU=24.04 docker compose run -it --rm -e ARROW_BUILD_TYPE=DEBUG ubuntu-cpp-odbc /bin/bash

and execute the build, install driver and test using the same command specified on the compose.yaml file:

root@3dad19853bc0:/# /arrow/ci/scripts/cpp_build.sh /arrow /build
...
root@3dad19853bc0:/# sudo /arrow/cpp/src/arrow/flight/sql/odbc/install/unix/install_odbc.sh /usr/local/lib/libarrow_flight_sql_odbc.so
Adding [Apache Arrow Flight SQL ODBC Driver] to odbcinst.ini...
root@3dad19853bc0:/# /arrow/ci/scripts/cpp_test.sh /arrow /build
...
100% tests passed, 0 tests failed out of 56

Label Time Summary:
arrow-compute-tests = 102.32 sec*proc (15 tests)
arrow-tests = 82.14 sec*proc (38 tests)
arrow_flight = 7.66 sec*proc (2 tests)
arrow_flight_sql = 1.44 sec*proc (1 test)
filesystem = 1.90 sec*proc (1 test)
unittest = 193.55 sec*proc (56 tests)

I can't seem to reproduce no matter how many times I try.

I have tried to debug and understand the crash but I haven't been able to advance much on it.

[alinaliBQ]

Thanks for looking into this Raul.

I was able to reproduce the error one time by running the docker compose run ... command you posted. Before I was trying to reproduce with the interactive shell as well and couldn't get the error to show no matter how many times I run the arrow-odbc-spi-impl-test test. Since the error can't be reproduced on the bash shell, I am not sure if this is an real issue or the tests react differently with ctest.

I think for now we could re-run the CI to get the workflow to pass. We will note this for future ODBC Linux CI runs.

cc @lidavidm @kou

[lidavidm]

FWIW, another way to debug may be to reproduce it on a fork and then use https://github.com/mxschmitt/action-tmate to poke at the binary

Or maybe we can get the core dump?

[alinaliBQ]

I have updated the description with logs from arrow-flight-sql-odbc-test which provides more information. I’ll defer to folks with more experience in gRPC area for diagnosis