[Tracking] Arrow Flight SQL and ADBC Catalog: issues and improvements

[morningman]

This issue tracks Arrow Flight SQL-related problems in Apache Doris. If you encounter a problem, please open a new issue or leave a comment here.

Goal

Use this issue as the single index for user-visible bugs, protocol gaps, reliability work, and ecosystem improvements related to Doris Arrow Flight SQL and the ADBC or Flight SQL clients that use it.

ADBC is the client API and driver layer. Doris implements the Arrow Flight SQL server path that those clients call. This tracker covers both sides when a client report identifies a Doris interoperability or server behavior problem.

Foundational context: #25514 and #19812.

Status convention

• Open, unmerged, closed without a verified fix, duplicated but not closed, or otherwise needs follow-up.
• The referenced change was merged or the work item was confirmed complete. Backports are listed beside their primary PR.

Open or needs follow-up

Reliability, lifecycle, and correctness

• External-table split lifecycle across GetFlightInfo and DoGet: [Bug] Arrow Flight SQL + Iceberg: SplitSource released before DoGet (fetchSplitBatch crash) #62259 is closed and the root-cause fix [fix](arrow-flight) Keep coordinator alive across GetFlightInfo/DoGet for external table scan #64799 merged on master on 2026-07-18, so only release-branch coverage is missing: there is no branch-4.1 or branch-4.0 backport of [fix](arrow-flight) Keep coordinator alive across GetFlightInfo/DoGet for external table scan #64799. [fix](arrow-flight) Harden split source error path to avoid BE crash on external table scan #64797 and its 4.1 backport branch-4.1: [fix](arrow-flight) Harden split source error path to avoid BE crash on external table scan #64797 #64890 harden the error path so the BE does not crash; they explicitly do not keep the SplitSource alive. Check this item once [fix](arrow-flight) Keep coordinator alive across GetFlightInfo/DoGet for external table scan #64799 reaches the supported branches.
• Corrupted large string/JSON results: [Bug] Arrow Flight SQL: Corrupted StringArray serialization results in empty DataBuffer and invalid Offsets #61124 reports invalid StringArray offsets and an empty data buffer.
• DDL regression on 4.0.5: [Bug][Arrow Flight SQL] DDL statements via Flight SQL crash with getMysqlChannel RuntimeException after 4.0.5 CLIENT_DEPRECATE_EOF fix #62017. Proposed fix [fix](arrow-flight) Fix DDL via Flight SQL crashing with getMysqlChannel RuntimeException #62018 was closed without merge. Reproduce on supported 4.0/4.1 builds and land or replace the fix.
• BE crash with public_host and an ILB proxy: [Bug] BE segfault in Arrow Flight SQL DoGet when using public_host (ILB proxy) #62217. Separate the endpoint-routing problem from the Arrow/gRPC status-conversion crash path and verify both.
• LargeUtf8 offset handling can crash remote Doris reads: the only proposed fix, PR [bug](arrow) fix arrow type change cause read data coredump #62681, was closed without merge on 2026-07-22, and no issue tracks the crash itself. Reproduce on a supported build and file a report, or land a replacement fix.
• Complex-type fidelity: [fix](arrow-flight) Preserve array item nullability #64976 proposed preserving array child nullability and [test](be) Add Arrow array smallint layout tests #65044 added focused IPC tests, but both closed without merge. [test](regression) skip Arrow-incompatible complex outputs #65182 documents additional Arrow Map null-key and raw BITMAP incompatibilities by skipping those regression cases. Decide which cases require engine fixes and which require documented protocol limitations.
• Flight SQL JDBC RST_STREAM failures on 4.0.1: [Bug] Arrow Flight SQL Jdbc driver fails with RST_STREAM on Doris 4.0.1 #59304 was closed as stale without a linked fix. Reproduce against current supported versions or link the resolving change.
• Flight SQL with the distributed planner disabled: [Bug] FlightSQL throws IllegalArgumentException: null 4.0无法使用flightsql #57318 was closed as stale. [fix](fe) Fix remote Flight SQL result receiver initialization #63136 appears to address the same empty-result-receiver failure on master and branch-4.1: [fix](fe) Fix remote Flight SQL result receiver initialization #63136 #63155 backports it to 4.1, but the 4.0 backport branch-4.0: [fix](fe) Fix remote Flight SQL result receiver initialization #63136 #63154 closed without merge. Verify the version matrix and close the loop explicitly.
• Sessions remain in SHOW PROCESSLIST after an ADBC connection closes: [Bug] Arrow flight SQL process hanging after closing connection #36331 was closed as stale. [fix](connection)Fixes session-related data is not cleared when the connection is disconnected due to timeout. #55008/branch-3.1:[fix](connection)Fixes session-related data is not cleared when the connection is disconnected due to timeout. #55008 #55396 improve timeout cleanup and [fix](connection) Prevent timeout checker from stopping after an exception #65040 keeps the timeout checker alive (4.1 backport branch-4.1: [fix](connection) Prevent timeout checker from stopping after an exception #65040 #65063 merged, 4.0 backport branch-4.0: [fix](connection) Prevent timeout checker from stopping after an exception #65040 #65062 still open), but explicit client-close behavior still needs a current-version verification.

Arrow type fidelity and client-visible behavior

Reported from an end-to-end Python ADBC sweep against 4.1.3-rc02 (commit 31263df4dc), with MySQL/JDBC used as the control path. Each item lists a workaround, so they are usable as documented limitations until they are fixed.

• DATETIME was returned as a timezone-aware Arrow timestamp: [Bug] ArrowFlight SQL result automatically converts timezone naive date to UTC (which is wrong) #65741. The naive-DATETIME work listed below appears to fix this on master and branch-4.1, but no PR declared that it closes the report. Verify on a current build and close it, or record what still reproduces.
• Raw HLL and QUANTILE_STATE reads fail with INTERNAL: [Bug] Arrow Flight SQL: reading raw HLL / QUANTILE_STATE columns fails with INTERNAL (IllegalStateException) #67364. StmtExecutor.sendFields() fails a Preconditions.checkState instead of returning a clean unsupported-type error. Same family as the raw BITMAP case skipped in [test](regression) skip Arrow-incompatible complex outputs #65182.
• Year-zero DATE values are off by one day: [Bug] Arrow Flight SQL: year-zero DATE values are encoded one day late #67366. 0000-01-01 arrives as 0000-01-02 over Arrow while JDBC is correct; looks like a proleptic-Gregorian versus Julian mismatch in the date32 day computation.
• First VARIANT read on a pre-existing connection loses empty objects: [Bug] Arrow Flight SQL: first VARIANT read on a pre-existing connection returns an empty string instead of {} #67367. A connection opened before the table was created returns "" instead of {} on the first read only; the second read on the same connection is correct.
• Short-circuit point query returns no Flight endpoint: [Bug] Arrow Flight SQL: short-circuit point query returns no Flight endpoint (no FlightSqlEndpointsLocations) #67368. fetch arrow flight schema failed, no FlightSqlEndpointsLocations; the same query succeeds with enable_short_circuit_query=false, so the short-circuit path appears not to register a Flight endpoint at all.
• PL/SQL CALL errors but executes its DML side effect four times: [Bug] Arrow Flight SQL: PL/SQL CALL returns INTERNAL and executes its DML side effect four times #67369. Returns getMysqlChannel not in mysql connection and still inserts four rows for a single call. Data-correctness impact; clients must not auto-retry on this error.
• UTC TIMESTAMPTZ uses the non-portable Arrow timezone Z: [Bug] Arrow Flight SQL: UTC TIMESTAMPTZ is tagged with the non-portable Arrow timezone "Z" #67370. PyArrow resolves the timezone string through zoneinfo and raises ZoneInfoNotFoundError; the epoch values themselves are correct.
• Iceberg BINARY is declared as Arrow string with a non-UTF-8 payload: [Bug] Arrow Flight SQL: Iceberg BINARY column is declared as Arrow string but carries invalid UTF-8 #67371. With enable.mapping.varbinary=false the declared type and the payload encoding disagree, so PyArrow fails the whole batch on UnicodeDecodeError.
• Doris MAP with a NULL key cannot be returned as an Arrow MAP: [Bug] Arrow Flight SQL: MAP with a NULL key cannot be returned as an Arrow MAP #67372. DoGet fails with Can not write null value of map key to arrow; map_entries() round-trips the same value, which suggests list<struct<key, value>> as one possible representation. Complements the skip added in [test](regression) skip Arrow-incompatible complex outputs #65182.

Authentication, endpoint advertisement, and session semantics

• Duplicate authentication reports: FlightSQL Unable to authenticate user #61743, FlightSQL Unable to authenticate user #61744, and doris FlightSQL Unable to authenticate user #61757 all report authentication using user@0.0.0.0. The fix [fix](auth)Fix arrow flight client ip auth #63506 and both backports branch-4.0: [fix](auth)Fix arrow flight client ip auth #63506 #63591/branch-4.1: [fix](auth)Fix arrow flight client ip auth #63506 #63592 are merged, but all three reports are still open. Verify the merged fix against the reported read-only user case, then close or deduplicate them.
• FE cannot advertise a public Flight endpoint: [Bug] When arrow-flight is used and the FE private name name is not resolvable, the client cannot access the FE nodes #62538 requests FE equivalents of BE public_host and the Flight proxy port.
• FE/BE advertises 127.0.0.1:8050 to a remote JDBC client: [Bug] Connection refused: no further information: /127.0.0.1:8050 JDK1.8 connect error #59490 was closed as stale. Check whether it is the same endpoint-advertisement gap as [Bug] When arrow-flight is used and the FE private name name is not resolvable, the client cannot access the FE nodes #62538.
• Kubernetes and reverse-proxy topology: [Feature] Improving arrow flight sql with Kubernetes setup #44599 is an older, stale-closed precursor to [Bug] BE segfault in Arrow Flight SQL DoGet when using public_host (ILB proxy) #62217/[Bug] When arrow-flight is used and the FE private name name is not resolvable, the client cannot access the FE nodes #62538. Decide whether the remaining work belongs in Doris, Doris Operator, or both, and cross-link the owner issue.
• Doris Catalog session-variable passthrough in Arrow Flight mode: [fix](doris catalog)Support session passthrough for Doris catalog #59742.

Protocol coverage and architecture

• Variant over cross-cluster Arrow Flight: [Bug] Cross-cluster Catalog with Arrow Flight fails to read Variant type #61883. The discussion proposes a LargeUtf8 first step and leaves a more efficient representation as follow-up.
• Prepared-statement parameter binding: apache/arrow-java#1009 was closed upstream as a duplicate, with the gap attributed to the Doris server side. Create or link a Doris issue that defines the required Flight SQL parameter schema and execution behavior.
• Initial database/schema in a Flight SQL JDBC URL: apache/arrow-java#829 remains open. Determine client-versus-server ownership and document the supported connection option.
• ADBC ingestion: users requested adbc_ingest/DataFrame writes in [Feature] Doris support Arrow Flight SQL protocol #25514, but no dedicated Doris work item tracks scope, protocol methods, atomicity, or type coverage. Create one if ingestion remains on the roadmap.
• Protocol SPI extraction: [feat](protocol)extract protocol definitions (part 1) #60355.
• Optional BE Arrow Flight worker pool: [opt](arrow_flight) Add enable_arrow_flight configuration option #60388 adds an enable_arrow_flight switch so unused deployments do not allocate the pool.

Tests and ecosystem

• Arrow regression cleanup: [test](regression) Avoid selecting raw bitmap column in Arrow Flight suite #62536 remains open.
• Arrow JDBC close-time allocator leak: [fix](regression) Avoid prepared Arrow JDBC path in remote IP auth test #64024 changed the authentication regression test to avoid the prepared path, and [improvement](regression) Align regression framework arrow/netty with fe-core and target JDK17 #64677 explicitly says the intermittent close-time leak needs a separate fix. Create or link that follow-up and check whether newer Arrow/Netty versions resolve it.
• Spark column projection: apache/doris-spark-connector#333 reports that doris.read.fields is ignored in Arrow read mode.
• Spark array work cleanup: apache/doris-spark-connector#345 remains open although the narrower opt-in implementation #362 merged. Mark the older PR superseded or document the remaining element-type work.

Completed or merged during the review window

Core feature and query path

• Doris Catalog and cross-cluster query over Arrow Flight: [Feature] Support Doris Catalog #55274 and [Feature] doris cross-cluster query #57898.
• Connection cleanup and thread-local correctness: [fix](connection)Fixes session-related data is not cleared when the connection is disconnected due to timeout. #55008 with 3.1 backport branch-3.1:[fix](connection)Fixes session-related data is not cleared when the connection is disconnected due to timeout. #55008 #55396, plus [fix](connection)Fix kill connection will make current connection killed #55809.
• Non-destructive Flight ticket/query-id parsing: [fix] remove parse_id #57240 with 4.0 backport branch-4.0: [fix] remove parse_id #57240 #57330.
• Flight processor close ordering and noisy warning fix: [fix][arrow-flight-sql]Move ctx.clear() to the end of close method #57858 with 4.0 backport branch-4.0: [fix][arrow-flight-sql]Move ctx.clear() to the end of close method #57858 #58011, closing [Bug] many warn logs about arrow-flight-sql processor #57859.
• Queries containing & no longer produce malformed tickets: fix flight sql when contains & #58830 with 4.0 backport branch-4.0: fix flight sql when contains & #58830 #58884.
• Async result completion avoids dirty reads: [bugfix](arrowflight) should call done run in on_xxx method to make work in async mode #60282 with 4.0 backport branch-4.0: [bugfix](arrowflight) should call done run in on_xxx method to make work in async mode #60282 #60325.
• Remote Flight SQL result receiver initialization: [fix](fe) Fix remote Flight SQL result receiver initialization #63136 with 4.1 backport branch-4.1: [fix](fe) Fix remote Flight SQL result receiver initialization #63136 #63155.
• Arrow UTF8/String size limit corrected to the intended 2 GiB threshold: [fix](be) Correct Arrow UTF8/String size limit #63137 with backports branch-4.1: [fix](be) Correct Arrow UTF8/String size limit #63137 #63148 and branch-4.0: [fix](be) Correct Arrow UTF8/String size limit #63137 #63558.
• CommandGetTables schema and the TIMESTAMPTZ Arrow reader corrected: [fix](arrow) correct the Flight SQL GetTables schema and the TIMESTAMPTZ arrow reader #66344 with 4.1 backport branch-4.1: [fix](arrow) correct the Flight SQL GetTables schema and the TIMESTAMPTZ arrow reader #66344 #66355. The FE GetTables mapping had drifted from what BE actually writes.
• DATETIME is returned as a timezone-naive Arrow timestamp so clients stop reading a wall-clock value as an instant: [fix](exec) Return DATETIME as naive Arrow timestamp (#10492) #66873 with 4.1 backport [fix](exec) Return DATETIME as naive Arrow timestamp  #67027, plus the Thrift path in [fix](exec) Return Thrift DATETIME as naive Arrow timestamp #67232 with 4.1 backport branch-4.1: [fix](exec) Return Thrift DATETIME as naive Arrow timestamp #67232 #67299. The earlier attempt [fix](exec) Return DATETIME as naive Arrow timestamp #65823 was closed without merge. This is the likely fix for [Bug] ArrowFlight SQL result automatically converts timezone naive date to UTC (which is wrong) #65741, which is still open.
• ADBC catalog that reads an external source over Arrow: [feat](catalog) support ADBC catalog that reads external sources over Arrow #66331. Adds an adbc catalog type so rows arrive as Arrow record batches without per-value JNI conversion and a scan is split across backends. Master only; no backport.
• Timeout checking continues after one connection throws: [fix](connection) Prevent timeout checker from stopping after an exception #65040 with 4.1 backport branch-4.1: [fix](connection) Prevent timeout checker from stopping after an exception #65040 #65063. The 4.0 backport branch-4.0: [fix](connection) Prevent timeout checker from stopping after an exception #65040 #65062 is still open.

Authentication, diagnostics, and crash hardening

• Better malformed-ticket logging: log print flight sql ticket error #58811 with 4.0 backport branch-4.0: log print flight sql ticket error #58811 #58842.
• Workload-group values are populated in audit records for early query failures: [fix](audit log) fix empty workload_group value __internal_schema.audit_log #62651 with 4.1 backport branch-4.1: [fix](audit log) fix empty workload_group value __internal_schema.audit_log #62651 #62697.
• Arrow status initialization crash fix: [Fix](arrow flight) Fix arrow::Status inline static empty msg core #63191 with backports branch-4.0: [Fix](arrow flight) Fix arrow::Status inline static empty msg core (#63191) #63266/branch-4.1: [Fix](arrow flight) Fix arrow::Status inline static empty msg core (#63191) #63267. The build-mode root cause was also addressed by [Fix](thirdparty) Unify arrow cpp standard with be #64390 and documented upstream in apache/arrow#49781.
• Real client IP is used for Flight authentication: [fix](auth)Fix arrow flight client ip auth #63506 with backports branch-4.0: [fix](auth)Fix arrow flight client ip auth #63506 #63591/branch-4.1: [fix](auth)Fix arrow flight client ip auth #63506 #63592.
• Bearer tokens are no longer written to fe.log in cleartext: [fix](arrow-flight) Stop writing bearer tokens to fe.log #66572 with 4.1 backport branch-4.1: [fix](arrow-flight) Stop writing bearer tokens to fe.log #67146. A token is a complete credential for arrow_flight_token_alive_time_second (86400s by default), so any deployment shipping FE logs off-box wants this; there is no 4.0 backport yet.
• External split-fetch failures now fail gracefully instead of crashing the BE: [fix](arrow-flight) Harden split source error path to avoid BE crash on external table scan #64797 with 4.1 backport branch-4.1: [fix](arrow-flight) Harden split source error path to avoid BE crash on external table scan #64797 #64890. This is only the robustness half of [Bug] Arrow Flight SQL + Iceberg: SplitSource released before DoGet (fetchSplitBatch crash) #62259; the lifecycle item remains open above.
• FE direct-memory leak during prepared-statement creation fixed: [fix](arrow-flight-sql) Close temporary VectorSchemaRoot in createPreparedStatement to fix FE direct memory leak #65311 (closes [Bug] Arrow Flight SQL causes FE Direct Memory leak until OOM ("Cannot reserve ... direct buffer memory", Internal; Prepare) #65305) closes the temporary VectorSchemaRoots in createPreparedStatement and adds a repeat-prepare regression test asserting the Flight SQL channel allocator returns to zero. The earlier attempt [fix](arrowflight) some resource unrelease bug #52245 was closed without merge. Backports have since merged: 4.1 branch-4.1: [fix](arrow-flight-sql) Close temporary VectorSchemaRoot in createPreparedStatement to fix FE direct memory leak #65311 #65751 and 4.0 branch-4.0: [fix](arrow-flight-sql) Close temporary VectorSchemaRoot in createPreparedStatement to fix FE direct memory leak #65311 #66431.

Regression framework and ecosystem

• Remote-IP auth regression uses a stable JDBC statement path: [fix](regression) Avoid prepared Arrow JDBC path in remote IP auth test #64024 with backports branch-4.0: [fix](regression) Avoid prepared Arrow JDBC path in remote IP auth test #64024 #64047/branch-4.1: [fix](regression) Avoid prepared Arrow JDBC path in remote IP auth test #64024 #64049.
• Regression framework Arrow/Netty versions aligned with FE and JDK 17: [improvement](regression) Align regression framework arrow/netty with fe-core and target JDK17 #64677.
• JDBC complex values are materialized before result-set teardown: [fix](regression) Materialize JDBC array results #65084.
• Regression cases now account for Arrow-incompatible Map null keys and protocol-specific BITMAP bytes: [test](regression) skip Arrow-incompatible complex outputs #65182.
• Spark connector FE/BE load balancing: apache/doris-spark-connector#329.
• Spark connector Flight read logging: apache/doris-spark-connector#354.
• Spark connector opt-in native array reads: apache/doris-spark-connector#362, closing #341.
• Flink connector BackendClient typo that could affect Flight reads: apache/doris-flink-connector#622.
• Website corrected the Flink connector source.use-flight-sql default for current/4.x docs: apache/doris-website#3725.

Closed without merge or superseded attempts

Keep these for history so the same investigation is not repeated:

• Prepared-statement resource release: [fix](arrowflight) some resource unrelease bug #52245.
• Query profile and connection finalization: [fix](arrow-flight-sql) Fix arrow flight sql query profile and correct close #45416.
• Early Arrow regression experiments: [test] fix arrow flight regression-test #53284, [test] fix arrow flight regression-test 2 #53285, and [test] fix arrow flight regression-test 3 #53286.
• Alternative generic Arrow Flight Catalog implementation: [Feature](arrow_flight catalog) support arrow_flight catalog #59270. The merged Doris Catalog path is [Feature] Support Doris Catalog #55274/[Feature] doris cross-cluster query #57898.
• Logging experiment: test LOGs for arrow flight #61990.
• DDL regression fix: [fix](arrow-flight) Fix DDL via Flight SQL crashing with getMysqlChannel RuntimeException #62018.
• 4.0 result-receiver backport: branch-4.0: [fix](fe) Fix remote Flight SQL result receiver initialization #63136 #63154.
• Array child nullability and focused IPC tests: [fix](arrow-flight) Preserve array item nullability #64976 and [test](be) Add Arrow array smallint layout tests #65044.
• LargeUtf8 offset fix attempt: [bug](arrow) fix arrow type change cause read data coredump #62681.
• Early naive-DATETIME attempt: [fix](exec) Return DATETIME as naive Arrow timestamp #65823, superseded by [fix](exec) Return DATETIME as naive Arrow timestamp (#10492) #66873.

Maintenance rules

• Add each new issue or PR under the closest topic and link it back to this tracker.
• Keep the primary fix and all required release-branch backports on the same line.
• Check an item only after the fix is merged and the affected supported versions are verified.
• If a stale-closed report still reproduces, reopen it or create a replacement and link both directions.
• Record user-visible protocol limitations in the Arrow Flight SQL documentation, not only in regression-test skips.

[wushilin]

@morningman
ADBC has timestamp conversion issue, tz naive timestamp is converted to UTC by default, which is wrong.

See #65741