[Bug] Arrow Flight SQL: an abandoned session holds its query's workload-group queue slot until wait_timeout (8h)

[raghav-reglobe]

Search before asking

• I had searched in the issues and found no similar issues.

Version

master (current head 219c619) and the 3.x/4.x lineage since #64799.

What's Wrong?

Since #64799 the FE keeps a BE-served Flight SQL query's coordinator alive after GetFlightInfo so the BE can still fetch external-table splits during DoGet. The coordinator is finalized only when the session runs its next query or is closed (CloseSession, idle kill, token expiry). While it lives it holds the query's workload-group queue slot and its information_schema.active_queries registration.

A Flight session idles under the same wait_timeout as a MySQL connection (28800s by default). Most Flight clients open a session per query and never send CloseSession (the ADBC and JDBC drivers do not send it on connection close, as the comment in DorisFlightSqlProducer.closeSession notes). So each abandoned session pins one queue slot for up to eight hours.

With max_concurrency=8 on the group, eight finished queries were enough: they stayed RUNNING in active_queries for over an hour and every later query in the group failed with query queue timeout, timeout: 60000 ms. SELECT 1-class queries kept working because they never enter the queue, which makes the lockout look intermittent.

What You Expected?

An abandoned Flight session should not hold a queue slot for hours. Either the slot is released once the result has been fetched, or Flight sessions get an idle bound much shorter than wait_timeout — without ever cutting a query short of its own query_timeout.

How to Reproduce?

1. CREATE WORKLOAD GROUP wg_t PROPERTIES ('max_concurrency'='2','max_queue_size'='5','queue_timeout'='10000'), grant it to a user and set it as their default workload group.
2. From a Flight SQL client, twice: handshake → GetFlightInfo on a scanning query (any table) → DoGet the result → do NOT close the session (or exit the process without CloseSession).
3. On the FE: SELECT * FROM information_schema.active_queries — both finished queries are still RUNNING; SHOW PROCESSLIST shows two Sleep sessions with Host = 0.0.0.0:0.
4. Run a third scanning query in that group from any protocol: it queues and fails after 10s with query queue timeout.
5. KILL CONNECTION <id> on either Flight session: the slot is released immediately.

Anything Else?

The MySQL protocol path releases the slot when the result is sent (finalizeCommand), so only Flight is affected. On the Doris side the natural fix is a Flight-specific idle bound — a new FE config applied by the existing connection timeout checker, floored at the query's exec timeout so a long DoGet drain is never killed early. Clients should still send CloseSession.

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[Doris-Breakwater]

Breakwater-GitHub-Analysis-Slot: slot_ca94d2a74982

Initial triage

Assessment: confirmed code-level lifecycle bug on master at 219c6193f24, with high operational impact for Arrow Flight SQL users whose workload groups have finite max_concurrency. The reported behavior is a direct consequence of the coordinator deferral added by #64799, rather than an unexplained runtime anomaly. No labels are currently set; please add the repository's Bug and Arrow Flight SQL/FE component labels if available.

Verified facts

• Every BE-served Flight query using the normal coordinator is registered in QeProcessorImpl, then marked deferred. Its eager coordBase.close() is skipped. The code explicitly notes that this broad gate covers internal as well as external remote-result queries and retains the queue slot/query registration until later cleanup (StmtExecutor, deferral, skipped close).
• The processor changes the connection to COM_SLEEP, deliberately skips finalization for deferred executors, and then clears ConnectContext.executor (FlightSqlConnectProcessor.close). Cleanup therefore occurs only on the next query, a failed GetFlightInfo, or connection teardown (DorisFlightSqlProducer, error path, pool teardown).
• A sleeping connection is checked only against session wait_timeout; the default is 28,800 seconds. The same startTime was set when the query began, and BE-served DoGet is directed to a BE endpoint, so it does not refresh FE activity (timeout check, BE endpoint construction and existing callback TODO).
• Coordinator.close() is the operation that releases the workload-group queue token, while finalizeArrowFlightQuery() closes the coordinator and unregisters the query. Retaining the deferred executor therefore explains both the occupied slot and the stale active_queries row (queue release, finalization). It also explains why KILL CONNECTION releases both immediately.
• The SELECT 1 observation is consistent with the implementation: queries without a non-schema scan node do not enter this workload-group queue, and short-circuit/point queries are not deferred by the coordBase == coord gate.
• The direct BE DoGet path currently only reads/forwards batches; there is no notification back to the FE when the stream reaches EOS (BE DoGetStatement, batch EOS). Thus "release when fetched" requires a new BE-to-FE completion mechanism; it cannot be implemented solely in the existing FE stream handler.

I verified this path by source inspection at the reported commit. I did not run the end-to-end reproduction in this environment.

Version scope correction

The deferral path is present at the current heads of master, branch-4.0, and branch-4.1. It is not present at the current heads of branch-3.0 or branch-3.1; those branches still eagerly close/finalize the coordinator at the end of GetFlightInfo. Unless a different 3.x branch/tag contains a separate backport, the issue's “3.x/4.x lineage” statement should be narrowed to master/4.x.

Missing information

No additional logs or profile are required to accept the lifecycle bug: the supplied reproduction and the current code are sufficient. For an automated integration test, it would still help to record the exact ADBC/JDBC driver and version used, plus whether all DoGet batches were consumed before the process exited. A Doris query profile is not needed for this resource-lifetime issue.

Recommended next steps

1. Treat a Flight-specific cleanup bound as the practical short-term fix, reusing the existing connection timeout checker and the existing unregisterConnection cleanup path. The timeout must be measured from the query's recorded start and must be at least that deferred query's immutable execution timeout.
2. Do not calculate that floor from ConnectContext.getExecTimeoutS() after GetFlightInfo: FlightSqlConnectProcessor.close() clears ConnectContext.executor, and per-query SET_VAR(query_timeout=...) values are reverted after statement execution. Capture the coordinator's TQueryOptions.execution_timeout (or an absolute deadline) when the executor is deferred, so the cleanup checker cannot terminate a long query based on a restored/default session value.
3. Longer term, implement the callback already noted in DorisFlightSqlProducer so BE execution/EOS can finalize the FE coordinator promptly. Keep the timeout as a fallback for abandoned/stalled consumers and lost callbacks.
4. Add coverage for: (a) two abandoned Flight sessions no longer blocking a third scanning query past the configured bound; (b) a long DoGet not being cleaned before its captured query deadline, including a SET_VAR(query_timeout=...) case; (c) immediate cleanup on CloseSession/KILL CONNECTION; (d) concurrent timeout/next-query teardown remaining single-finalization; and (e) master plus affected 4.x branches.