Double free in mariadb

[Octogonapus]

I ran into a double free. My program is using execute(db, ...) on multiple threads but it's guarded by a lock so no two executes can run concurrently. I've also experienced a few crashes which seem related (just segfaults with no printout).

MySQL v1.4.6
MariaDB_Connector_C_jll v3.1.12+0

double free or corruption (fasttop)

[2167724] signal (6.-6): Aborted
in expression starting at REPL[48]:1
pthread_kill at /lib/x86_64-linux-gnu/libc.so.6 (unknown line)
raise at /lib/x86_64-linux-gnu/libc.so.6 (unknown line)
abort at /lib/x86_64-linux-gnu/libc.so.6 (unknown line)
unknown function (ip: 0x7f0132879675)
unknown function (ip: 0x7f0132890cfb)
unknown function (ip: 0x7f01328929c9)
free at /lib/x86_64-linux-gnu/libc.so.6 (unknown line)
SSL_free at /home/salmon/.julia/artifacts/69903d1a57355b6e78750aac3a8b3fddab87180f/lib/libssl.so (unknown line)
ma_tls_close at /home/salmon/.julia/artifacts/c688df815e86a92f5f36c990983dd26d8a80f17a/lib/mariadb/libmariadb.so (unknown line)
ma_pvio_close at /home/salmon/.julia/artifacts/c688df815e86a92f5f36c990983dd26d8a80f17a/lib/mariadb/libmariadb.so (unknown line)
mthd_my_send_cmd at /home/salmon/.julia/artifacts/c688df815e86a92f5f36c990983dd26d8a80f17a/lib/mariadb/libmariadb.so (unknown line)
mysql_stmt_execute at /home/salmon/.julia/artifacts/c688df815e86a92f5f36c990983dd26d8a80f17a/lib/mariadb/libmariadb.so (unknown line)
mysql_stmt_execute at /home/salmon/.julia/packages/MySQL/NMNbH/src/api/ccalls.jl:8 [inlined]
execute at /home/salmon/.julia/packages/MySQL/NMNbH/src/api/papi.jl:3 [inlined]
#execute#35 at /home/salmon/.julia/packages/MySQL/NMNbH/src/prepare.jl:162
execute at /home/salmon/.julia/packages/MySQL/NMNbH/src/prepare.jl:152
unknown function (ip: 0x7eff73f23349)
_jl_invoke at /cache/build/builder-amdci4-4/julialang/julia-release-1-dot-10/src/gf.c:2895 [inlined]
ijl_apply_generic at /cache/build/builder-amdci4-4/julialang/julia-release-1-dot-10/src/gf.c:3077
#82 at /home/salmon/git/project/main.jl:70
lock at ./lock.jl:229
unknown function (ip: 0x7eff73fed249)
_jl_invoke at /cache/build/builder-amdci4-4/julialang/julia-release-1-dot-10/src/gf.c:2895 [inlined]
ijl_apply_generic at /cache/build/builder-amdci4-4/julialang/julia-release-1-dot-10/src/gf.c:3077
#81 at /home/salmon/git/project/main.jl:69
unknown function (ip: 0x7eff73fea5b2)
_jl_invoke at /cache/build/builder-amdci4-4/julialang/julia-release-1-dot-10/src/gf.c:2895 [inlined]
ijl_apply_generic at /cache/build/builder-amdci4-4/julialang/julia-release-1-dot-10/src/gf.c:3077
jl_apply at /cache/build/builder-amdci4-4/julialang/julia-release-1-dot-10/src/julia.h:1982 [inlined]
start_task at /cache/build/builder-amdci4-4/julialang/julia-release-1-dot-10/src/task.c:1238
Allocations: 121462440 (Pool: 121313001; Big: 149439); GC: 60
[1] 2167724 IOT instruction (core dumped) julia -t auto --project

julia> versioninfo()
Julia Version 1.10.5
Commit 6f3fdf7b362 (2024-08-27 14:19 UTC)
Build Info:
 Official https://julialang.org/ release
Platform Info:
 OS: Linux (x86_64-linux-gnu)
 CPU: 32 × 13th Gen Intel(R) Core(TM) i9-13900K
 WORD_SIZE: 64
 LIBM: libopenlibm
 LLVM: libLLVM-15.0.7 (ORCJIT, goldmont)
Threads: 32 default, 0 interactive, 16 GC (on 32 virtual cores)

kernel 5.15.153.1-microsoft-standard-WSL2

[quinnj]

I hit this same crash and dug into it — I believe I have the root cause, a fast reproducer, and controls that isolate the trigger.

TL;DR: the API.MYSQL_STMT finalizer calls mysql_stmt_close, which is not a client-side free — it sends COM_STMT_CLOSE over the connection's socket. Finalizers run on whichever thread triggers GC, and GC (plus finalizers) runs while other threads are inside mysql_* ccalls. So an abandoned prepared statement's finalizer does socket I/O on a MYSQL* concurrently with a legitimate call on another thread — no user-level lock can prevent it, because the user's lock isn't (and can't be) held by GC. libmariadb connections are not thread-safe, so this is memory-unsafe.

The finalizer in question (src/api/apitypes.jl#L56-L61):

finalizer(stmt) do x
 if x.ptr != C_NULL
 mysql_stmt_close(x.ptr) # sends COM_STMT_CLOSE over the wire
 x.ptr = C_NULL
 end
end

API.MYSQL (mysql_close sends COM_QUIT) and API.MYSQL_RES (mysql_free_result reads any un-fetched rows off the wire) have the same problem in their finalizers.

Caught in the act

macOS crash report from the abort, showing two threads simultaneously inside libmariadb on the same connection — thread 3 is a GC finalizer closing a statement, thread 5 is the application thread (which held a ReentrantLock serializing every DB call):

Thread 3:
 libsystem_c.dylib abort
 libsystem_malloc.dylib malloc_vreport / malloc_zone_error / free_tiny_botch <- double free
 libssl.3.dylib ossl_ssl_connection_free
 libssl.3.dylib SSL_free
 libmariadb.3.dylib ma_tls_close
 libmariadb.3.dylib ma_pvio_close
 libmariadb.3.dylib end_server
 libmariadb.3.dylib mthd_my_send_cmd
 libmariadb.3.dylib net_stmt_close
 libmariadb.3.dylib mysql_stmt_close
 libjulia-internal jl_gc_run_finalizers_in_list <- GC finalizer
 libjulia-internal run_finalizers / ijl_gc_collect / maybe_collect / jl_gc_small_alloc_inner

Thread 5 (faulting):
 libsystem_c.dylib abort
 libsystem_malloc.dylib malloc_vreport / malloc_zone_error / free_tiny_botch <- double free
 libssl.3.dylib SSL_CTX_free
 libssl.3.dylib SSL_free
 libmariadb.3.dylib ma_tls_close
 libmariadb.3.dylib ma_pvio_close
 libmariadb.3.dylib ma_net_safe_read
 libmariadb.3.dylib mthd_my_read_query_result
 libmariadb.3.dylib mysql_commit <- application call, lock held

The interleaved use of one SSL* from two threads corrupts OpenSSL state; then both threads' error paths call ma_pvio_close → ma_tls_close → SSL_free on the same SSL — the double free in this issue's title. Across runs I've seen the abort land in SSL_write (under mysql_stmt_execute), in SSL_free/SSL_CTX_free (as above and as in the OP's trace: mysql_stmt_execute → mthd_my_send_cmd → ma_pvio_close → ma_tls_close → SSL_free), and once as a straight SIGSEGV jumping through a freed pointer.

Why it looks TLS-correlated

COM_STMT_CLOSE is fire-and-forget (no server response). On a plaintext socket, the racing finalizer write usually just interleaves at packet granularity and nothing visibly breaks — no shared client-side state is being mutated concurrently except the net buffer. Under TLS, both threads mutate the same SSL record layer (cipher sequence, buffers), so you get (2026): TLS/SSL error: ssl/tls alert bad record mac, wedged connections, or heap corruption → SIGABRT/SIGSEGV. The race is present without TLS too — one plaintext run here produced ArgumentError: unsafe_wrap: pointer 0x9 is not properly aligned to 8 bytes from a result fetch, i.e. silent client-side corruption instead of a crash.

Reproducer

Crashes within seconds-to-a-minute on every run here (3/3 runs ended in SIGABRT or SIGSEGV; interspersed rounds die with bad record mac). All DB access is behind a single ReentrantLock, as in the OP.

# julia -t 8 repro.jl (against e.g.: docker run -d -p 3306:3306 \
# -e MYSQL_ALLOW_EMPTY_PASSWORD=yes -e MYSQL_DATABASE=test mysql:8)
using MySQL, DBInterface

const PORT = 3306
myconnect() = DBInterface.connect(MySQL.Connection, "127.0.0.1", "root", "";
 db="test", port=PORT)

let conn = myconnect()
 DBInterface.execute(conn, "DROP TABLE IF EXISTS finalizer_race")
 DBInterface.execute(conn, "CREATE TABLE finalizer_race (k INT PRIMARY KEY, v INT)")
 DBInterface.execute(conn, "INSERT INTO finalizer_race VALUES (1, 0)")
 DBInterface.close!(conn)
end

function hammer(conn, secs)
 lk = ReentrantLock() # serializes ALL user access to conn
 done = Threads.Atomic{Bool}(false)
 firsterr = Ref{Any}(nothing)
 @sync begin
 for _ in 1:6
 Threads.@spawn try
 while !done[]
 lock(lk) do
 # idiomatic prepared-statement use; statements abandoned
 # (no close!) -> garbage -> finalizer sends COM_STMT_CLOSE
 # from whatever thread runs GC
 stmt = DBInterface.prepare(conn,
 "UPDATE finalizer_race SET v = v + 1 WHERE k = 1")
 DBInterface.execute(stmt)
 stmt2 = DBInterface.prepare(conn,
 "SELECT v FROM finalizer_race WHERE k = 1")
 DBInterface.execute(stmt2)
 end
 end
 catch e
 firsterr[] === nothing && (firsterr[] = e)
 done[] = true
 end
 end
 Threads.@spawn while !done[] # GC pressure: finalizers run on this thread
 GC.gc(false)
 end
 Threads.@spawn (t0 = time(); while !done[] && time() - t0 < secs; sleep(0.2); end; done[] = true)
 end
 return firsterr[]
end

for r in 1:10
 conn = myconnect()
 err = hammer(conn, 20)
 println("round $r: ", err === nothing ? "clean" : sprint(showerror, err))
 try DBInterface.close!(conn) catch end
end

Controls

configuration	result
TLS + abandoned statements (above)	SIGABRT/SIGSEGV every run (plus bad record mac rounds)
TLS + DBInterface.close!(stmt) inside the lock	no crash (4×20s rounds clean)
plaintext (server with tls_version=) + abandoned statements	no crash, but one round died with unsafe_wrap: pointer 0x9 is not properly aligned

The explicit-close control is the smoking gun: the only difference is whether mysql_stmt_close runs under the user's lock or in a GC finalizer.

Single-threaded Julia (-t 1) is safe — finalizers can't run concurrently with a ccall on the same thread. Any -t ≥ 2 workload that ever lets a Statement, un-drained MYSQL_RES, or Connection be GC'd while the connection is in use is exposed, which is why this shows up as rare, unexplainable crashes in long test suites.

Environment

• macOS 15.6.1 aarch64 (also reproduces the OP's Linux x86_64 trace shape exactly)
• Julia 1.12.6, 8 threads
• MySQL.jl 1.5.1 (bug present on current main, 1.5.2) / MariaDB_Connector_C_jll 3.4.9+1 / OpenSSL_jll 3.5.4
• Server: mysql:8 (8.4.11) in Docker

Two aggravating factors worth noting:

1. With MariaDB_Connector_C_jll 3.4.x, TLS is on by default for every connection that talks to a TLS-capable server, and ssl_mode=SSL_MODE_DISABLED cannot turn it off (that option maps onto a different, unrelated option in libmariadb — filing separately). So essentially all users of current MySQL.jl + mysql:8 are in the crashy configuration.
2. On macOS the @rpath/libssl.3.dylib reference resolves to Julia's bundled lib/julia/libssl.3.dylib rather than OpenSSL_jll's — didn't affect this bug, but it means the OpenSSL actually in use isn't the one the jll pins.

Fix sketch

The invariant the fix needs: finalizers must never touch the socket. One workable shape:

• Give Connection a small reap queue (spinlock-guarded Vector{Ptr{Cvoid}}). The MYSQL_STMT finalizer pushes the raw ptr and returns — no libmariadb call.
• Drain the queue (calling mysql_stmt_close on each ptr) at the start of every user-facing operation on that connection (prepare, execute, transaction, close!…). Those sites are already serialized by whatever discipline the user applies, so the close packets go out under the same protection as everything else.
• DBInterface.close!(conn) drains the queue before mysql_close. If the connection is already closed/dead, drop the ptrs via plain mysql_stmt_close after mysql_close has invalidated them (libmariadb turns that into a local free), or leak them — either is better than a crash.
• Same treatment for MYSQL_RES (an un-drained result's mysql_free_result reads from the socket) and the MYSQL finalizer's mysql_close.

Happy to turn this into a PR if the approach sounds right.