[feat](catalog) support ADBC catalog that reads external sources over Arrow by morningman · Pull Request #66331 · apache/doris

morningman · 2026-07-31T09:20:05Z

What problem does this PR solve?

Related Issue: #65615

Problem Summary:

Doris reads an external database through JDBC today: the source's rows become JDBC
objects and then Doris columns, one value at a time, across JNI. The source is also
read by a single backend, because JDBC has no notion of how a result is partitioned.

This PR adds a new catalog type, adbc, that reads an external source through an
Arrow Database Connectivity driver. Data arrives as
Arrow record batches and is handed to BE's scanner without a per-value conversion, and
a scan is split into the driver's own result partitions so that N backends read N
partitions in parallel. Neither is reachable through JDBC at all.

CREATE CATALOG remote_source PROPERTIES (
"type"="adbc",
"driver_url"="libadbc_driver_flightsql.so",
"uri"="grpc://remote-doris:8070",
"user"="root",
"password"=""
);
SELECTcount(*) FROMremote_source.some_db.some_table;

Phase one targets Arrow Flight SQL sources, which makes this the intended replacement
for the remote_doris catalog. Other sources need only their driver's .so and, where
their SQL differs from ANSI, a dialect implementation -- no architectural change.

Supported features

What an ADBC catalog gives you from the outside -- what you can do with it and what it does
for you without being asked. The boundaries are in Known limits further down.

Capability	What it means when you use it	Control (default)
ADBC catalog	`CREATE CATALOG ... PROPERTIES("type" = "adbc", ...)`, then query the source's tables like any other external table. Phase one targets Arrow Flight SQL sources, including another Doris cluster.	`uri`, `user`, `password`
Arrow-native transfer	Rows arrive as Arrow batches and go straight into the scanner, with no per-value conversion on the way in -- which is where a JDBC catalog spends much of a wide or high-volume scan.	Always on
Parallel read across backends	One scan is split into the source's own result partitions and read by several BEs at once, instead of a single BE on a single connection. Falls back to one statement when the source cannot partition.	`partitioned_read` = `auto` / `disabled` / `required` (`auto`), `max_partitions` (`1024`)
Column pruning	Only the columns the query actually needs are requested from the source, and `EXPLAIN` prints that same statement.	Always on
Predicate pushdown	`=`, `!=`, `<`, `<=`, `>`, `>=`, `IS [NOT] NULL`, `[NOT] IN` and `AND` / `OR` / `NOT` over them become the remote `WHERE`. Anything else -- functions, arithmetic, `LIKE`, `BETWEEN` -- stays in Doris. Doris re-applies every predicate regardless, so pushdown changes speed only, never the rows you get.	Always on
`LIMIT` pushdown	Pushed once the whole `WHERE` was pushed, so the source never truncates ahead of a filter Doris still has to apply.	Always on
`COUNT(*)` without column values	A count reads no column data from the source at all.	Always on
Metadata browsing	`SHOW DATABASES`, `SHOW TABLES`, `DESC`, `SHOW CREATE TABLE` and `information_schema` over the source's own databases, tables and columns.	--
Automatic type mapping	Source columns arrive as Doris types, including `ARRAY` / `MAP` / `STRUCT`, `DECIMAL`, date, and timestamp with and without a zone.	--
Metadata cache and `REFRESH`	Name resolution and table schemas are remembered per catalog, so a query stops paying several remote round trips before it is even planned. Listings are still read live, so a table created on the source is visible with no refresh at all, and `REFRESH CATALOG` / `DATABASE` / `TABLE` drops what is remembered.	`meta.cache.adbc.metadata.enable` / `.ttl-second` (`600`) / `.capacity` (`1000`)
The usual query surface	Joins against internal tables and against other catalogs, aggregation, `ORDER BY`, `UNION`, subqueries, `SELECT ... INTO OUTFILE`, and an MTMV built on an ADBC table.	--
Source SQL you can steer	The generated SQL is conservative ANSI. The connector asks the driver which vendor it is talking to, and you can override that when the answer is unhelpful.	`sql_dialect` (auto-detected, `ansi` fallback; `doris` provided)
Driver options passed through	Anything the driver itself understands can be set on the catalog, e.g. `"adbc.adbc.snowflake.sql.db" = "..."`.	`adbc.*` properties
Driver placement and pinning	You place the driver library (Doris ships none); a bare file name resolves under the drivers directory, and a wrong or stale build is reported at `CREATE CATALOG` rather than as a puzzling query failure later.	`driver_url`, `driver_checksum`, `driver_entrypoint`

Design decisions worth knowing before reviewing

FE and BE load the same driver .so. FE goes through the ADBC JNI bridge, which
wraps the same C driver manager BE links statically, and BE dlopens the identical
file. This is a measured constraint, not a preference: partition descriptors are
driver-private bytes, and two official implementations of the same protocol already
serialize incompatible messages and mis-parse each other silently rather than
erroring.
Doris ships no ADBC driver. The library is placed by the operator, under
adbc_drivers_dir on FE and be/plugins/adbc_drivers on every BE. driver_url
therefore accepts local references only (bare name, file://, or an absolute path);
remote schemes are rejected because a per-node download cannot promise the nodes
agree. "Driver file not found" is written as a first-class error for that reason, and
driver_checksum can pin the build.
The SQL sent to a source is conservative ANSI by default, generated through a
dialect interface a source can claim by vendor name or by the sql_dialect property.
Predicates are pushed all-or-nothing per conjunct from a whitelist; BE re-applies
every predicate regardless, so pushdown is pure acceleration.
Planning has side effects on a Flight SQL source: asking the driver to partition a
statement executes it. That makes this the first connector for which EXPLAIN had to
be told not to plan for real -- see the SPI addition below.

Changes outside the connector

Change	Why
`ConnectorScanRequest.isExplainOnly()` (SPI) + `PluginDrivenScanNode` fills it	`EXPLAIN` reaches `planScan` for real (its explain level is `NORMAL`, so `NereidsPlanner.distribute()` does not return early). For this connector that would execute the very query it was asked only to describe.
`PluginDrivenScanNode` re-asks for the display statement with the current columns	The connector properties are cached in `init()`, before Nereids prunes the scan tuple, so `EXPLAIN`'s `QUERY:` line named more columns than the statement that actually runs. Affects any connector that renders remote SQL from the column list (adbc, jdbc). What goes to BE is unchanged.
`PluginDrivenScanNode.mapFileFormatType()` learns `"arrow"`	Routes an ADBC range to BE's Arrow reader.
`TTableFormatFileDesc.adbc_params`	The scan range's parameters; BE's ADBC reader does not read the JDBC field.
`be/src/vec/exec/format/adbc/*` + `file_scanner_v2` gate	The BE reader.
`FlightSqlSchemaHelper` (FE, Arrow Flight SQL server)	`GetTables` described a `DATEV2` column as date64 while BE writes date32, and described array/map/struct with placeholder children. A client that types its columns from that schema fails on the first batch. Visible to every Arrow Flight SQL client, not only this connector.
`DataTypeTimeStampTzSerDe::read_column_from_arrow` (BE)	It never overrode the numeric serde's fixed-width path, which memcpy'd Arrow's int64 epochs into a column that stores packed date/time values -- same width, so every row was silently wrong.
`thirdparty`: `arrow-adbc`	The C driver manager is linked into `doris_be`; the JNI bridge is built for FE (upstream's prebuilt one needs GLIBC 2.34). The SQLite and Flight SQL drivers are fetched for tests only and are not shipped.
`Config.adbc_drivers_dir` / `adbc_driver_secure_path`, `conf/fe.conf` JVM options, `build.sh`	Driver placement and the FE-side plugin build/deploy wiring.

Behaviour to be aware of

A view on the source is not listed as a table. A Doris source ignores the base-table
filter ADBC sends, so the filter is applied where the answer is read.
partitioned_read is auto by default: split the scan when the driver can, read it
as one statement when it cannot. required forbids that downgrade, which is what
keeps a test from going green while quietly exercising the fallback; disabled is the
escape hatch.
Metadata is cached per catalog for 10 minutes by default
(meta.cache.adbc.metadata.*), and every REFRESH statement drops it. A newly
created remote table is reachable without any refresh.
Read only: INSERT, CREATE/DROP TABLE and the other write statements against an
ADBC catalog are rejected. An MTMV over an ADBC table does build and refresh.

Test coverage

fe-connector-adbc: 186 unit tests, no skips. About 25 of them drive the real SQLite
ADBC driver through the same Java -> JNI -> C driver manager -> driver .so path FE
takes in production; fe-connector-api covers the new SPI field.
BE: unit tests for the ADBC reader, the driver registry, the Arrow variant normalizer,
the scanner gate and the TIMESTAMPTZ serde.
End-to-end (regression-test/suites/external_table_p0/adbc/): 20 suites against a live
catalog -- type mapping and semantics, complex and binary types, scan edges, predicate
pushdown, column pruning, query shapes, source table models, metadata operations,
nested catalogs, cross-source joins, large data, outfile, MTMV, negative cases, and
partitioned vs. single-statement reads compared row for row. Each suite returns early
when the driver is absent, so a cluster without one does not fail.

Known limits

Read only. No writes, DDL, statistics or aggregate pushdown.
Partitions are verified to be read completely and without duplication, but spreading
them over several backends has only been reasoned from the code, not observed -- the
test environment has one backend, and test_adbc_multi_backend returns early there.
IPV4 from a Doris source arrives as a bare INT (both sides encode it as int32), and
a source DATETIME arrives as TIMESTAMPTZ. Both are asserted, not worked around.
The FE and BE copies of the driver are not checked against each other;
driver_checksum verifies FE's copy only.

Release note

Support ADBC catalog, which reads an external source through an Arrow Database
Connectivity driver: data is transferred as Arrow record batches and a scan is split
across backends by the driver's own result partitions. Phase one targets Arrow Flight
SQL sources. The driver library is supplied by the operator; Doris ships none.

Also fixes two defects on the Arrow Flight SQL path that are visible to existing
clients: GetTables reported the wrong Arrow type for DATEV2 and placeholder children
for ARRAY/MAP/STRUCT, and a TIMESTAMPTZ column read from Arrow was decoded by
copying its bits instead of converting the epoch.

Check List (For Author)

Test
- Regression test
- Unit Test
- Manual test (add detailed scripts or steps below)
- No need to test or manual test. Explain why:
  - This is a refactor/code format and no logic has been changed.
  - Previous test can cover this change.
  - No code files have been changed.
  - Other reason
Behavior changed:
- No.
- Yes.
  - New catalog type adbc; no existing catalog type changes.
  - Arrow Flight SQL clients now see the Arrow types the batches actually carry for
    DATEV2 and for nested types (previously a client that trusted GetTables
    failed on the first batch).
  - EXPLAIN on a plugin-driven catalog prints the projection the scan really asks
    for; the statement sent to BE is unchanged.
Does this need documentation?
- No.
- Yes.

Check List (For Reviewer who merge this PR)

Confirm the release note
Confirm test cases
Confirm document
Add branch pick label

hello-stephen · 2026-07-31T09:20:12Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

What problem was fixed (it's best to include specific error reporting information). How it was fixed.
Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
What features were added. Why was this function added?
Which code was refactored and why was this part of the code refactored?
Which functions were optimized and what is the difference before and after the optimization?

morningman · 2026-08-02T05:52:05Z

run buildall

hello-stephen · 2026-08-02T06:29:47Z

Cloud UT Coverage Report

Increment line coverage 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	77.57% (1912/2465)
Line Coverage	64.49% (34197/53024)
Region Coverage	64.37% (17257/26809)
Branch Coverage	53.91% (9244/17148)

hello-stephen · 2026-08-02T07:13:35Z

FE UT Coverage Report

Increment line coverage 13.79% (4/29) 🎉
Increment coverage report
Complete coverage report

hello-stephen · 2026-08-02T10:54:10Z

BE UT Coverage Report

Increment line coverage 80.16% (501/625) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	59.23% (25691/43373)
Line Coverage	43.33% (258032/595471)
Region Coverage	39.02% (204632/524487)
Branch Coverage	40.35% (93367/231391)

hello-stephen · 2026-08-02T12:17:43Z

BE Regression && UT Coverage Report

Increment line coverage 80.32% (502/625) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	75.80% (32068/42304)
Line Coverage	60.49% (357770/591495)
Region Coverage	57.14% (300720/526254)
Branch Coverage	58.48% (135336/231429)

hello-stephen · 2026-08-02T12:58:46Z

FE Regression Coverage Report

Increment line coverage 34.48% (10/29) 🎉
Increment coverage report
Complete coverage report

### What problem does this PR solve? Related Issue: #65615 Related PR: #66331 Problem Summary: Split out of #66331, which adds an `adbc` catalog type that reads an external source through an [Arrow Database Connectivity](https://arrow.apache.org/adbc/) driver. This PR carries only that PR's `thirdparty/` half, so the dependency can be reviewed and the build-env image rebuilt before the code that links against it lands. Nothing in the tree consumes these artifacts yet -- this PR adds one package to the thirdparty build and declares its license, and changes nothing else. **What comes out of it** | Artifact | How | Used by | |---|---|---| | `libadbc_driver_manager.a` | built from source | to be statically linked into `doris_be` (#66331) | | `libadbc_driver_jni.so` | built from source | to be loaded by the FE ADBC connector (#66331) | | `libadbc_driver_sqlite.so` | built from source | tests only, **not shipped** | | `libadbc_driver_flightsql.so` | prebuilt, from the official release wheel | tests only, **not shipped** | Doris ships no ADBC driver to users; a deployment supplies its own. The two drivers above exist so the ADBC code paths can be tested at all. **Three things upstream does that do not carry over** - *The SQLite driver needs a system SQLite3 development package*, which Doris does not ship and most build hosts lack. The source tree vendors the amalgamation but never references it from CMake, so it is compiled here into a scratch static library, handed to `FindSQLite3`, and dropped afterwards. It ends up statically inside the driver, leaving no sqlite artifacts in thirdparty. - *The JNI bridge header is generated by shelling out to Maven* (`java/driver/jni/CMakeLists.txt` runs `mvn -Pjni,javah compile`). Doing that would make this the first thirdparty package to require Maven, a Maven Central connection and a JDK 11+, while the build-env image runs this script with `JAVA_HOME` on JDK 8. The `javah` output is checked in as a patch instead and `jni_wrapper.cc` is compiled against it directly, needing nothing but `jni.h`. The patch header records how to regenerate it on a version bump. The prebuilt JNI binary inside upstream's Maven jar is not used either: it requires `GLIBC_2.34` and `GLIBCXX_3.4.31`, which excludes CentOS 7/8, Rocky 8 and Ubuntu 20.04. - *The Flight SQL driver is written in Go and no bare shared library is published*, so it is taken from the official release wheel -- a zip the existing download step already knows how to unpack -- rather than adding a Go toolchain to the thirdparty build. It is skipped on platforms upstream publishes no prebuilt binary for, the same way hyperscan is. **On the version pin** The source tree is tag `apache-arrow-adbc-24`, which is release C/Go 1.12.0 (the tag carries neither number). The prebuilt Flight SQL driver is pinned to that same release, and that is not cosmetic: ADBC partition descriptors are driver-private bytes, so every process that hands one to another must have loaded the very same driver build. `dist/LICENSE-dist.txt` gets the corresponding Apache-2.0 entry. It is the only file outside `thirdparty/` here.

morningman · 2026-08-03T01:51:41Z

run buildall

hello-stephen · 2026-08-03T03:27:26Z

FE UT Coverage Report

Increment line coverage 13.79% (4/29) 🎉
Increment coverage report
Complete coverage report

hello-stephen · 2026-08-03T07:08:20Z

TPC-H: Total hot run time: 28798 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 4315a843d1343d982e241375a9c865a6adbba9ab, data reload: false
------ Round 1 ----------------------------------
============================================
q1	17689	3902	3934	3902
q2	2036	323	196	196
q3	10833	1406	826	826
q4	4805	477	340	340
q5	8416	869	571	571
q6	345	166	133	133
q7	810	805	597	597
q8	10596	1587	1560	1560
q9	5637	4065	4064	4064
q10	6796	1648	1347	1347
q11	526	353	316	316
q12	741	590	447	447
q13	18123	3277	2780	2780
q14	266	257	234	234
q15	q16	736	723	657	657
q17	915	873	945	873
q18	6771	5698	5588	5588
q19	1175	1217	1114	1114
q20	825	702	596	596
q21	5559	2599	2361	2361
q22	417	357	296	296
Total cold run time: 104017 ms
Total hot run time: 28798 ms
----- Round 2, with runtime_filter_mode=off -----
============================================
q1	4247	4173	4155	4155
q2	286	314	200	200
q3	4578	4939	4381	4381
q4	2192	2273	1405	1405
q5	4227	4112	4112	4112
q6	227	172	127	127
q7	1720	1834	1655	1655
q8	2439	2101	2022	2022
q9	7352	7144	7197	7144
q10	4355	4265	3891	3891
q11	558	395	377	377
q12	721	756	508	508
q13	3345	3459	2910	2910
q14	310	324	286	286
q15	q16	681	721	643	643
q17	1318	1274	1279	1274
q18	7925	7389	6883	6883
q19	1080	1039	1027	1027
q20	2214	2258	1916	1916
q21	5228	4605	4383	4383
q22	521	481	405	405
Total cold run time: 55524 ms
Total hot run time: 49704 ms

hello-stephen · 2026-08-03T07:19:07Z

TPC-DS: Total hot run time: 169203 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 4315a843d1343d982e241375a9c865a6adbba9ab, data reload: false
query5	4313	613	470	470
query6	453	214	203	203
query7	4935	592	323	323
query8	333	184	160	160
query9	8791	4106	4073	4073
query10	486	349	311	311
query11	5794	2194	2010	2010
query12	148	97	96	96
query13	1235	634	440	440
query14	6197	4753	4392	4392
query14_1	3839	3841	3855	3841
query15	213	206	178	178
query16	1038	489	465	465
query17	1136	722	536	536
query18	2438	450	336	336
query19	202	185	141	141
query20	104	102	99	99
query21	236	152	131	131
query22	13070	12983	12785	12785
query23	17328	16346	15930	15930
query23_1	16110	16171	16073	16073
query24	7485	1729	1225	1225
query24_1	1243	1257	1232	1232
query25	530	414	352	352
query26	1322	362	210	210
query27	2585	619	395	395
query28	4470	2065	1994	1994
query29	1078	604	486	486
query30	340	259	225	225
query31	1108	1070	942	942
query32	103	61	58	58
query33	506	297	245	245
query34	1196	1132	639	639
query35	744	735	622	622
query36	787	789	701	701
query37	153	100	85	85
query38	1843	1650	1623	1623
query39	834	816	781	781
query39_1	783	785	820	785
query40	249	164	146	146
query41	69	61	61	61
query42	92	91	88	88
query43	312	315	274	274
query44	1417	778	761	761
query45	188	172	166	166
query46	1041	1165	716	716
query47	1530	1511	1434	1434
query48	405	409	294	294
query49	594	412	302	302
query50	1081	452	337	337
query51	10784	10853	10703	10703
query52	87	86	78	78
query53	257	269	202	202
query54	297	232	212	212
query55	71	71	65	65
query56	291	288	283	283
query57	1013	1005	920	920
query58	289	277	249	249
query59	1500	1550	1364	1364
query60	298	265	252	252
query61	175	175	166	166
query62	400	327	268	268
query63	239	209	205	205
query64	2936	1103	974	974
query65	3886	3853	3794	3794
query66	1851	498	374	374
query67	28198	28174	28045	28045
query68	3276	1548	955	955
query69	408	295	276	276
query70	873	803	767	767
query71	376	331	320	320
query72	3183	2724	2245	2245
query73	866	771	432	432
query74	4649	4475	4258	4258
query75	2406	2361	1996	1996
query76	2315	1136	743	743
query77	336	363	278	278
query78	11142	11064	10523	10523
query79	1476	1117	737	737
query80	1280	530	463	463
query81	531	328	287	287
query82	619	151	114	114
query83	385	327	294	294
query84	286	152	134	134
query85	1004	591	508	508
query86	409	230	230	230
query87	1790	1796	1739	1739
query88	3759	2811	2788	2788
query89	398	318	284	284
query90	1922	214	204	204
query91	203	189	159	159
query92	63	60	54	54
query93	1681	1483	936	936
query94	720	379	321	321
query95	783	565	471	471
query96	1129	800	365	365
query97	2437	2430	2312	2312
query98	202	212	188	188
query99	719	717	607	607
Total cold run time: 256898 ms
Total hot run time: 169203 ms

hello-stephen · 2026-08-03T07:23:55Z

ClickBench: Total hot run time: 25.15 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 4315a843d1343d982e241375a9c865a6adbba9ab, data reload: false
query1	0.01	0.00	0.01
query2	0.15	0.08	0.09
query3	0.37	0.24	0.25
query4	1.61	0.25	0.25
query5	0.33	0.32	0.32
query6	1.17	0.67	0.66
query7	0.04	0.00	0.00
query8	0.09	0.08	0.08
query9	0.50	0.38	0.37
query10	0.57	0.57	0.60
query11	0.30	0.18	0.19
query12	0.31	0.19	0.19
query13	0.53	0.53	0.54
query14	0.93	0.90	0.91
query15	0.68	0.59	0.58
query16	0.39	0.40	0.38
query17	1.00	1.00	1.01
query18	0.29	0.28	0.28
query19	1.90	1.82	1.76
query20	0.02	0.02	0.02
query21	15.42	0.36	0.32
query22	4.91	0.15	0.14
query23	15.84	0.52	0.30
query24	2.42	0.63	0.46
query25	0.15	0.11	0.11
query26	0.76	0.28	0.22
query27	0.11	0.09	0.10
query28	3.47	0.83	0.42
query29	12.45	4.09	3.18
query30	0.44	0.27	0.26
query31	2.76	0.58	0.33
query32	3.23	0.60	0.47
query33	2.94	2.93	2.94
query34	15.66	3.92	3.26
query35	3.17	3.21	3.16
query36	0.63	0.52	0.51
query37	0.13	0.10	0.09
query38	0.08	0.07	0.06
query39	0.07	0.07	0.07
query40	0.21	0.17	0.18
query41	0.13	0.09	0.09
query42	0.09	0.06	0.06
query43	0.07	0.07	0.06
Total cold run time: 96.33 s
Total hot run time: 25.15 s

hello-stephen · 2026-08-03T07:30:10Z

BE Regression && UT Coverage Report

Increment line coverage 80.32% (502/625) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	75.79% (32084/42333)
Line Coverage	60.45% (357773/591889)
Region Coverage	57.00% (300133/526566)
Branch Coverage	58.33% (135078/231594)

hello-stephen · 2026-08-03T07:36:50Z

FE Regression Coverage Report

Increment line coverage 0.90% (10/1114) 🎉
Increment coverage report
Complete coverage report

Nothing references the ADBC symbols yet, so the archive contributes no objects to the binary until AdbcDriverRegistry lands. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

Doris serdes only accept the Arrow variants Doris itself emits: the string serde takes STRING, BINARY and FIXED_SIZE_BINARY and nothing else. Third-party ADBC drivers emit others -- DuckDB emits string_view, Go-based drivers may emit large_* and dictionary. Convert those before materialization, and fail with the offending type named when no Doris column can hold it, because silently wrong data is far worse than a loud error. Normalization loops rather than converting once: decoding dictionary<int32, large_utf8> leaves large_utf8 behind, which still needs converting. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

The driver manager dlopens on every AdbcDatabaseInit and keeps no cache of its own, so without this every scan range would reload the driver. Load each resolved path at most once and never dlclose: drivers carry global state and background threads -- Go runtimes especially -- so unloading one is a use-after-free hazard. Failed loads are cached too, so a bad path does not retry the dlopen once per scan range, and the failure message carries the path the user configured. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

Follows the string-map shape jdbc_params and es_params already established. The partition descriptor is opaque binary, so it travels base64-encoded rather than as a typed field. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

Mirrors remote_doris_reader: the Arrow-stream-to-Block materialization is identical, only the stream source changes from Arrow Flight to ADBC. Each column is normalized before materialization, since third-party drivers emit Arrow variants the serdes reject. The reader drives the driver's own function table rather than the driver manager's free functions. Those re-dlopen on every AdbcDatabaseInit, which would defeat AdbcDriverRegistry. Databases are not pooled across scan ranges. That is a throughput optimization which only pays off once multiple partitions run concurrently, and adding an untestable caching layer now would only obscure the functional path. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

ADBC only has a v2 reader, and enable_file_scanner_v2 is a fuzzy=true session variable the regression harness flips at random, so honoring it would make ADBC queries fail on a coin flip. Force adbc onto v2, mirroring the existing transactional_hive force-to-v1. Loads are excluded: there is no ADBC load path, so widening the rule to cover them would only route them somewhere they still cannot run. is_supported also has to accept adbc under FORMAT_ARROW, otherwise the scanner would refuse the very ranges the operator forces onto it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

Three things the unit tests cannot reach, because each needs the engine sitting in front of the connector. A table created behind Doris's back, by the sqlite3 CLI the fixture is already built with, must be queryable with no REFRESH at all. That works only if the connector answers the engine's re-list from the source; it is the end-to-end half of the rule, and it fails against a connector that serves that listing from memory. REFRESH CATALOG must reach the connector, asserted through a schema change rather than a new table: the engine clears its own schema copy either way, so a stale answer after the refresh can only have come from the connector. Without the hook the catalog would serve the schema it first read until the TTL expired, and REFRESH CATALOG does not rebuild the connector. The cache knobs, both directions: a catalog with the cache turned off still works, and an unparseable ttl fails at CREATE CATALOG. The second doubles as proof that the deployed plugin is the new one -- an older build ignores an unknown property, so the CREATE would succeed and the assertion would fail rather than pass quietly. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01F1z6M8Gj9F9pErafGdfdjx

driver_checksum has been a property name with nothing behind it since the connector was written: a user could set it and Doris would ignore it. It is now checked at CREATE CATALOG, by MD5, against the file this FE resolves. It earns its place from what this catalog type asks of an operator. Doris ships no ADBC driver, so the library is placed by hand on every node and the copies have to stay identical -- and a wrong or stale copy announces nothing. It loads, it answers, and whatever it does differently arrives later as a query failure that never mentions a file. The property remains optional and stays honest about its reach: it sees this FE's copy and no BE's, so it does not verify that the nodes agree. What it gives an operator is a way to state which build the catalog was written for and have one node check itself against that. A checksum that cannot be computed fails rather than passes, or the property would be quietly optional on exactly the node that could not read the file. Not routed through the validation context's own checksum service: that one resolves against jdbc_drivers_dir and enforces a .jar grammar, neither of which applies here. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01F1z6M8Gj9F9pErafGdfdjx

When a source refuses the pushed-down statement, its answer arrives in its own words -- typically a syntax error pointing at a quote character. Nothing in that says Doris wrote the statement, and nothing says which SQL Doris writes is a catalog property. That is the first wall anyone pointing this connector at something other than Doris walks into, because the default dialect is conservative ANSI and a source that wants something else rejects the very first query. The planning-time failure now names the dialect the statement was generated in and the property that changes it. Only the partitioned path can carry this: asking the driver to split a statement executes it, so FE is where that rejection lands. The single-statement path runs on BE and its error is BE's to improve. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01F1z6M8Gj9F9pErafGdfdjx

…ment A TIMESTAMPTZ literal reaches the dialect already converted to UTC: in a +08:00 session, ts > '2023-01-01 00:00:00' arrives as the wall clock 2022-12-31 16:00:00. Standard SQL's TIMESTAMP '...' spelling carries no zone, so the source reads that UTC wall clock as its own local time and compares it against its column. On a source east of UTC that only widens the match and Doris narrows it again on what comes back. On one west of UTC it NARROWS the match, and the rows the source drops are rows the query wanted -- a scan cannot ask for rows the source never sent. There is no portable spelling that says which instant is meant, so the comparison stays with Doris, for the same reason NaN, null-safe equality and LIKE already do. Also records why this connector does not consult the catalog property enable.mapping.timestamp_tz when mapping a zoned arrow timestamp: ExternalCatalog stamps that property as "false" into every external catalog that does not name it, so reading it cannot tell a user who asked for wall clocks from one who said nothing, and honouring it would force DATETIMEV2 on every adbc catalog. This connector's default is TIMESTAMPTZ; making the property settable needs fe-core to let a connector supply its own default first. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01F1z6M8Gj9F9pErafGdfdjx

Twenty suites over a live catalog: type mapping and semantics, complex and binary types, scan edges, predicate pushdown, column pruning, query shapes, source table models, metadata operations, nested catalogs, cross source joins, large data, outfile, MTMV, and negative cases. Several assertions are written against what the planner actually delivers rather than against what the SQL says, because Nereids normalises the predicate before any connector sees it: a one-column disjunction becomes an IN list, NOT is pushed through comparisons and connectives by De Morgan, column-plus-literal arithmetic is folded, and <=> against a non-null literal becomes =. Pinning the SQL spelling would have been asserting on the optimizer, so each of those keeps a shape the optimizer cannot rewrite -- a two-column OR, two-column arithmetic, two nullable columns of one type -- to hold the connector's own behaviour. Likewise count(*) is asserted as one narrow column, not zero: pruning an empty scan tuple puts the smallest slot back. Three expectations record losses this connector cannot avoid. IPV4 arrives as the address's 32 bits read signed, because Doris encodes it as int32 on both sides of the wire. A source datetime arrives as TIMESTAMPTZ, so comparisons cast it back before checking the instant. DBL_MAX cannot reach the test client at all -- Doris prints a double with 16 significant digits, which parses back as infinity -- so that row is compared inside Doris with a null-safe join instead. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01F1z6M8Gj9F9pErafGdfdjx

Both of this connector's deployment-level settings move off fe.conf and into the plugin's own adbc.conf -- the channel apache#66347 introduced -- as drivers_dir and driver_secure_path, read through ConnectorConf.get. Neither has an fe.conf half. The two @ConfFields they used to be (adbc_drivers_dir, adbc_driver_secure_path) and the two DefaultConnectorContext env entries that forwarded them are removed rather than kept as a fallback: this connector has never shipped, so no deployment configured them anywhere else, and a key in fe-core is an engine change per connector setting -- which is what that channel exists to stop. The default drivers directory is computed in the connector from the doris_home the engine already publishes, so it stays <DORIS_HOME>/plugins/adbc_drivers. build.sh needs no change: it seeds a live <name>.conf from any *.conf.template found in a plugin zip. AdbcConnectorConfTest pins the template's name against ConnectorProvider.name() -- a template under any other name deploys a file the engine never opens, with every setting in it silently ignored -- and pins that an environment still carrying the old fe.conf keys does not resurrect a channel fe-core no longer feeds. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01RgJjuW5w4jEKF9HorENTur

morningman · 2026-08-04T07:56:13Z

run buildall

hello-stephen · 2026-08-04T08:29:50Z

Cloud UT Coverage Report

Increment line coverage 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	77.57% (1912/2465)
Line Coverage	64.53% (34217/53024)
Region Coverage	64.44% (17275/26809)
Branch Coverage	53.97% (9255/17148)

hello-stephen · 2026-08-04T09:28:48Z

FE UT Coverage Report

Increment line coverage 16.67% (1/6) 🎉
Increment coverage report
Complete coverage report

hello-stephen · 2026-08-04T12:02:59Z

TPC-H: Total hot run time: 29211 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 1bc74d14f2e5ddffc7b24a2d47c58e896503c89f, data reload: false
------ Round 1 ----------------------------------
============================================
q1	17632	4034	4002	4002
q2	2006	324	202	202
q3	10318	1549	824	824
q4	4683	477	340	340
q5	7498	840	562	562
q6	189	174	141	141
q7	756	819	616	616
q8	9404	1627	1546	1546
q9	5700	4102	4074	4074
q10	6803	1657	1358	1358
q11	528	370	322	322
q12	756	575	459	459
q13	18084	3433	2856	2856
q14	263	261	238	238
q15	q16	742	726	672	672
q17	1046	1004	1009	1004
q18	6671	5650	5554	5554
q19	1166	1336	1021	1021
q20	800	690	582	582
q21	5643	2593	2526	2526
q22	425	365	312	312
Total cold run time: 101113 ms
Total hot run time: 29211 ms
----- Round 2, with runtime_filter_mode=off -----
============================================
q1	4434	4377	4270	4270
q2	282	333	218	218
q3	4602	5119	4424	4424
q4	2188	2270	1398	1398
q5	4354	4277	4254	4254
q6	233	181	130	130
q7	1757	1653	1415	1415
q8	2385	2013	2049	2013
q9	6784	6791	6760	6760
q10	4274	4269	3880	3880
q11	522	436	363	363
q12	696	714	501	501
q13	3079	3449	2878	2878
q14	284	298	275	275
q15	q16	684	701	614	614
q17	1251	1245	1233	1233
q18	12108	11021	11759	11021
q19	1076	1078	1080	1078
q20	2220	2212	1924	1924
q21	5434	4774	4678	4678
q22	535	480	417	417
Total cold run time: 59182 ms
Total hot run time: 53744 ms

hello-stephen · 2026-08-04T12:13:38Z

TPC-DS: Total hot run time: 166274 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 1bc74d14f2e5ddffc7b24a2d47c58e896503c89f, data reload: false
query5	4321	582	451	451
query6	462	224	195	195
query7	4846	598	351	351
query8	322	163	145	145
query9	8791	4060	4039	4039
query10	487	345	296	296
query11	5795	2155	1993	1993
query12	151	100	99	99
query13	1289	630	470	470
query14	6075	4293	4025	4025
query14_1	3878	3828	3823	3823
query15	206	202	179	179
query16	1020	518	400	400
query17	1121	718	564	564
query18	2573	483	340	340
query19	241	181	143	143
query20	105	100	100	100
query21	223	151	134	134
query22	13013	12962	12863	12863
query23	15883	15037	14524	14524
query23_1	14709	14705	14629	14629
query24	7578	1697	1243	1243
query24_1	1262	1263	1245	1245
query25	541	420	362	362
query26	1312	362	207	207
query27	2606	591	399	399
query28	4535	2066	2039	2039
query29	1042	624	469	469
query30	343	256	218	218
query31	1167	1111	1059	1059
query32	137	60	60	60
query33	503	309	231	231
query34	1187	1129	654	654
query35	744	737	630	630
query36	779	788	681	681
query37	160	106	105	105
query38	1843	1765	1661	1661
query39	820	839	801	801
query39_1	786	781	786	781
query40	247	160	140	140
query41	64	62	60	60
query42	92	91	90	90
query43	317	317	277	277
query44	1438	760	770	760
query45	202	174	166	166
query46	1020	1162	716	716
query47	1527	1531	1472	1472
query48	387	418	296	296
query49	581	400	298	298
query50	1151	416	340	340
query51	10742	10582	10437	10437
query52	85	87	74	74
query53	263	271	193	193
query54	275	228	213	213
query55	74	72	69	69
query56	301	284	287	284
query57	1024	1004	958	958
query58	291	251	261	251
query59	1494	1621	1397	1397
query60	307	265	247	247
query61	150	146	150	146
query62	387	320	268	268
query63	229	199	203	199
query64	2890	1166	961	961
query65	3863	3807	3834	3807
query66	1823	477	376	376
query67	28270	28059	27994	27994
query68	3469	1647	918	918
query69	429	323	270	270
query70	877	800	805	800
query71	423	320	318	318
query72	2997	2642	2278	2278
query73	832	839	441	441
query74	4651	4498	4304	4304
query75	2381	2365	1992	1992
query76	2424	1163	771	771
query77	344	361	276	276
query78	11266	11075	10684	10684
query79	1367	1150	730	730
query80	664	553	475	475
query81	448	326	287	287
query82	611	176	135	135
query83	405	339	301	301
query84	288	163	131	131
query85	954	620	537	537
query86	320	223	224	223
query87	1989	1968	1824	1824
query88	3715	2812	2807	2807
query89	396	318	279	279
query90	1932	196	186	186
query91	204	189	164	164
query92	64	60	57	57
query93	1590	1631	1078	1078
query94	551	339	312	312
query95	778	567	466	466
query96	1057	809	349	349
query97	2433	2450	2334	2334
query98	196	185	183	183
query99	743	729	615	615
Total cold run time: 253317 ms
Total hot run time: 166274 ms

hello-stephen · 2026-08-04T12:58:50Z

ClickBench: Total hot run time: 29.16 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 1bc74d14f2e5ddffc7b24a2d47c58e896503c89f, data reload: true
query1	0.00	0.00	0.01
query2	0.07	0.03	0.03
query3	0.22	0.13	0.13
query4	1.03	0.14	0.14
query5	0.23	0.23	0.22
query6	0.93	0.84	0.81
query7	0.03	0.01	0.00
query8	0.04	0.03	0.03
query9	0.42	0.31	0.32
query10	1.83	1.82	1.77
query11	0.20	0.13	0.13
query12	0.18	0.14	0.14
query13	0.54	0.53	0.53
query14	1.01	1.00	0.99
query15	0.68	0.65	0.65
query16	0.31	0.32	0.32
query17	1.15	1.10	1.09
query18	0.16	0.15	0.15
query19	2.12	1.96	1.92
query20	0.02	0.01	0.01
query21	11.26	0.28	0.13
query22	3.44	0.05	0.04
query23	11.82	0.41	0.11
query24	0.96	0.29	0.19
query25	1.32	0.31	0.13
query26	0.12	0.12	0.12
query27	0.14	0.13	0.12
query28	3.47	1.08	0.33
query29	9.18	4.60	3.23
query30	0.24	0.14	0.14
query31	1.85	0.64	0.32
query32	3.26	0.59	0.50
query33	3.29	3.18	3.24
query34	11.20	5.50	5.22
query35	5.31	5.26	5.26
query36	0.53	0.47	0.45
query37	0.08	0.06	0.06
query38	0.05	0.04	0.03
query39	0.04	0.03	0.02
query40	0.17	0.16	0.14
query41	0.08	0.02	0.02
query42	0.04	0.03	0.02
query43	0.05	0.03	0.03
Total cold run time: 79.07 s
Total hot run time: 29.16 s

hello-stephen · 2026-08-04T14:39:59Z

FE Regression Coverage Report

Increment line coverage 11.54% (3/26) 🎉
Increment coverage report
Complete coverage report

morningman · 2026-08-04T15:02:25Z

run check_coverage

github-actions · 2026-08-05T00:15:38Z

PR approved by at least one committer and no changes requested.

github-actions · 2026-08-05T00:15:41Z

PR approved by anyone and no changes requested.

## Summary Adds user documentation for the ADBC Catalog introduced in apache/doris#66331. An ADBC Catalog reads an external data source through an [Arrow Database Connectivity](https://arrow.apache.org/adbc/) driver. Data is transferred in Arrow format instead of being converted value by value as in a JDBC catalog, and one scan is split by the driver's own result partitions so several BE nodes read it in parallel. Phase one targets Arrow Flight SQL sources, including another Doris cluster. The feature is experimental and ships in 5.0.0. ## What the doc covers - Overview, differences from a JDBC Catalog, applicable scenarios and a feature matrix - Driver deployment. Doris ships no ADBC driver, and the FE resolves `driver_url` into an absolute path that it sends to the BEs unchanged, so the same driver build must sit at the same absolute path on the FE and on every BE. The `adbc.conf` settings (`drivers_dir`, `driver_secure_path`) are documented alongside. - Catalog properties, organized as an unordered list in the same style as the Iceberg doc: `driver_url`, `uri`, `{DriverProperties}`, `{ConnectionProperties}`, `{ReadProperties}`, `{DriverOptions}`, `{CommonProperties}` - Examples: querying another Doris cluster, pinning the driver build with `driver_checksum`, requiring parallel reads, disabling parallel reads - Namespace mapping from ADBC's three naming levels onto the two a Doris external table has - Column type mapping from Arrow types, including unsigned widening and the behavior for unmappable types - Query operations: column pruning, predicate pushdown, `LIMIT` pushdown, `COUNT(*)`, inspecting the generated remote SQL, parallel reads - Metadata cache (`meta.cache.adbc.metadata.*`), manual refresh and observability - Limitations and FAQ ## Notes - Both the English page and the Chinese translation are included, and only the `current` version is touched. - `sidebars.ts` lists the new page after `doris-catalog`, since the ADBC catalog is the intended replacement for it. - Content was verified against the source on the feature branch rather than the PR description alone. Two places where the two disagree follow the code: `user` and `password` are documented as optional, because only `driver_url` and `uri` are actually required; and `drivers_dir` / `driver_secure_path` are documented as `adbc.conf` settings rather than `fe.conf` keys. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: morningman <moringman@apache.org> Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>

### What problem does this PR solve? Related Issue: apache#65615 Related PR: apache#66331 Problem Summary: Split out of apache#66331, which adds an `adbc` catalog type that reads an external source through an [Arrow Database Connectivity](https://arrow.apache.org/adbc/) driver. This PR carries only that PR's `thirdparty/` half, so the dependency can be reviewed and the build-env image rebuilt before the code that links against it lands. Nothing in the tree consumes these artifacts yet -- this PR adds one package to the thirdparty build and declares its license, and changes nothing else. **What comes out of it** | Artifact | How | Used by | |---|---|---| | `libadbc_driver_manager.a` | built from source | to be statically linked into `doris_be` (apache#66331) | | `libadbc_driver_jni.so` | built from source | to be loaded by the FE ADBC connector (apache#66331) | | `libadbc_driver_sqlite.so` | built from source | tests only, **not shipped** | | `libadbc_driver_flightsql.so` | prebuilt, from the official release wheel | tests only, **not shipped** | Doris ships no ADBC driver to users; a deployment supplies its own. The two drivers above exist so the ADBC code paths can be tested at all. **Three things upstream does that do not carry over** - *The SQLite driver needs a system SQLite3 development package*, which Doris does not ship and most build hosts lack. The source tree vendors the amalgamation but never references it from CMake, so it is compiled here into a scratch static library, handed to `FindSQLite3`, and dropped afterwards. It ends up statically inside the driver, leaving no sqlite artifacts in thirdparty. - *The JNI bridge header is generated by shelling out to Maven* (`java/driver/jni/CMakeLists.txt` runs `mvn -Pjni,javah compile`). Doing that would make this the first thirdparty package to require Maven, a Maven Central connection and a JDK 11+, while the build-env image runs this script with `JAVA_HOME` on JDK 8. The `javah` output is checked in as a patch instead and `jni_wrapper.cc` is compiled against it directly, needing nothing but `jni.h`. The patch header records how to regenerate it on a version bump. The prebuilt JNI binary inside upstream's Maven jar is not used either: it requires `GLIBC_2.34` and `GLIBCXX_3.4.31`, which excludes CentOS 7/8, Rocky 8 and Ubuntu 20.04. - *The Flight SQL driver is written in Go and no bare shared library is published*, so it is taken from the official release wheel -- a zip the existing download step already knows how to unpack -- rather than adding a Go toolchain to the thirdparty build. It is skipped on platforms upstream publishes no prebuilt binary for, the same way hyperscan is. **On the version pin** The source tree is tag `apache-arrow-adbc-24`, which is release C/Go 1.12.0 (the tag carries neither number). The prebuilt Flight SQL driver is pinned to that same release, and that is not cosmetic: ADBC partition descriptors are driver-private bytes, so every process that hands one to another must have loaded the very same driver build. `dist/LICENSE-dist.txt` gets the corresponding Apache-2.0 entry. It is the only file outside `thirdparty/` here.

… Arrow (apache#66331) ### What problem does this PR solve? Related Issue: apache#65615 Problem Summary: Doris reads an external database through JDBC today: the source's rows become JDBC objects and then Doris columns, one value at a time, across JNI. The source is also read by a single backend, because JDBC has no notion of how a result is partitioned. This PR adds a new catalog type, `adbc`, that reads an external source through an [Arrow Database Connectivity](https://arrow.apache.org/adbc/) driver. Data arrives as Arrow record batches and is handed to BE's scanner without a per-value conversion, and a scan is split into the driver's own result partitions so that N backends read N partitions in parallel. Neither is reachable through JDBC at all. ```sql CREATE CATALOG remote_source PROPERTIES ( "type" = "adbc", "driver_url" = "libadbc_driver_flightsql.so", "uri" = "grpc://remote-doris:8070", "user" = "root", "password" = "" ); SELECT count(*) FROM remote_source.some_db.some_table; ``` Phase one targets Arrow Flight SQL sources, which makes this the intended replacement for the `remote_doris` catalog. Other sources need only their driver's `.so` and, where their SQL differs from ANSI, a dialect implementation -- no architectural change. ### Supported features What an ADBC catalog gives you from the outside -- what you can do with it and what it does for you without being asked. The boundaries are in *Known limits* further down. | Capability | What it means when you use it | Control (default) | |---|---|---| | ADBC catalog | `CREATE CATALOG ... PROPERTIES("type" = "adbc", ...)`, then query the source's tables like any other external table. Phase one targets Arrow Flight SQL sources, including another Doris cluster. | `uri`, `user`, `password` | | Arrow-native transfer | Rows arrive as Arrow batches and go straight into the scanner, with no per-value conversion on the way in -- which is where a JDBC catalog spends much of a wide or high-volume scan. | Always on | | Parallel read across backends | One scan is split into the source's own result partitions and read by several BEs at once, instead of a single BE on a single connection. Falls back to one statement when the source cannot partition. | `partitioned_read` = `auto` / `disabled` / `required` (`auto`), `max_partitions` (`1024`) | | Column pruning | Only the columns the query actually needs are requested from the source, and `EXPLAIN` prints that same statement. | Always on | | Predicate pushdown | `=`, `!=`, `<`, `<=`, `>`, `>=`, `IS [NOT] NULL`, `[NOT] IN` and `AND` / `OR` / `NOT` over them become the remote `WHERE`. Anything else -- functions, arithmetic, `LIKE`, `BETWEEN` -- stays in Doris. Doris re-applies every predicate regardless, so pushdown changes speed only, never the rows you get. | Always on | | `LIMIT` pushdown | Pushed once the whole `WHERE` was pushed, so the source never truncates ahead of a filter Doris still has to apply. | Always on | | `COUNT(*)` without column values | A count reads no column data from the source at all. | Always on | | Metadata browsing | `SHOW DATABASES`, `SHOW TABLES`, `DESC`, `SHOW CREATE TABLE` and `information_schema` over the source's own databases, tables and columns. | -- | | Automatic type mapping | Source columns arrive as Doris types, including `ARRAY` / `MAP` / `STRUCT`, `DECIMAL`, date, and timestamp with and without a zone. | -- | | Metadata cache and `REFRESH` | Name resolution and table schemas are remembered per catalog, so a query stops paying several remote round trips before it is even planned. Listings are still read live, so a table created on the source is visible with no refresh at all, and `REFRESH CATALOG` / `DATABASE` / `TABLE` drops what is remembered. | `meta.cache.adbc.metadata.enable` / `.ttl-second` (`600`) / `.capacity` (`1000`) | | The usual query surface | Joins against internal tables and against other catalogs, aggregation, `ORDER BY`, `UNION`, subqueries, `SELECT ... INTO OUTFILE`, and an MTMV built on an ADBC table. | -- | | Source SQL you can steer | The generated SQL is conservative ANSI. The connector asks the driver which vendor it is talking to, and you can override that when the answer is unhelpful. | `sql_dialect` (auto-detected, `ansi` fallback; `doris` provided) | | Driver options passed through | Anything the driver itself understands can be set on the catalog, e.g. `"adbc.adbc.snowflake.sql.db" = "..."`. | `adbc.*` properties | | Driver placement and pinning | You place the driver library (Doris ships none); a bare file name resolves under the drivers directory, and a wrong or stale build is reported at `CREATE CATALOG` rather than as a puzzling query failure later. | `driver_url`, `driver_checksum`, `driver_entrypoint` | ### Design decisions worth knowing before reviewing - **FE and BE load the same driver `.so`.** FE goes through the ADBC JNI bridge, which wraps the same C driver manager BE links statically, and BE dlopens the identical file. This is a measured constraint, not a preference: partition descriptors are driver-private bytes, and two official implementations of the same protocol already serialize incompatible messages and mis-parse each other silently rather than erroring. - **Doris ships no ADBC driver.** The library is placed by the operator, under `adbc_drivers_dir` on FE and `be/plugins/adbc_drivers` on every BE. `driver_url` therefore accepts local references only (bare name, `file://`, or an absolute path); remote schemes are rejected because a per-node download cannot promise the nodes agree. "Driver file not found" is written as a first-class error for that reason, and `driver_checksum` can pin the build. - **The SQL sent to a source is conservative ANSI by default**, generated through a dialect interface a source can claim by vendor name or by the `sql_dialect` property. Predicates are pushed all-or-nothing per conjunct from a whitelist; BE re-applies every predicate regardless, so pushdown is pure acceleration. - **Planning has side effects on a Flight SQL source**: asking the driver to partition a statement executes it. That makes this the first connector for which `EXPLAIN` had to be told not to plan for real -- see the SPI addition below. ### Changes outside the connector | Change | Why | |---|---| | `ConnectorScanRequest.isExplainOnly()` (SPI) + `PluginDrivenScanNode` fills it | `EXPLAIN` reaches `planScan` for real (its explain level is `NORMAL`, so `NereidsPlanner.distribute()` does not return early). For this connector that would execute the very query it was asked only to describe. | | `PluginDrivenScanNode` re-asks for the display statement with the current columns | The connector properties are cached in `init()`, before Nereids prunes the scan tuple, so `EXPLAIN`'s `QUERY:` line named more columns than the statement that actually runs. Affects any connector that renders remote SQL from the column list (adbc, jdbc). What goes to BE is unchanged. | | `PluginDrivenScanNode.mapFileFormatType()` learns `"arrow"` | Routes an ADBC range to BE's Arrow reader. | | `TTableFormatFileDesc.adbc_params` | The scan range's parameters; BE's ADBC reader does not read the JDBC field. | | `be/src/vec/exec/format/adbc/*` + `file_scanner_v2` gate | The BE reader. | | `FlightSqlSchemaHelper` (FE, Arrow Flight SQL server) | `GetTables` described a `DATEV2` column as date64 while BE writes date32, and described array/map/struct with placeholder children. A client that types its columns from that schema fails on the first batch. Visible to every Arrow Flight SQL client, not only this connector. | | `DataTypeTimeStampTzSerDe::read_column_from_arrow` (BE) | It never overrode the numeric serde's fixed-width path, which memcpy'd Arrow's int64 epochs into a column that stores packed date/time values -- same width, so every row was silently wrong. | | `thirdparty`: `arrow-adbc` | The C driver manager is linked into `doris_be`; the JNI bridge is built for FE (upstream's prebuilt one needs GLIBC 2.34). The SQLite and Flight SQL drivers are fetched for tests only and are **not shipped**. | | `Config.adbc_drivers_dir` / `adbc_driver_secure_path`, `conf/fe.conf` JVM options, `build.sh` | Driver placement and the FE-side plugin build/deploy wiring. | ### Behaviour to be aware of - A view on the source is not listed as a table. A Doris source ignores the base-table filter ADBC sends, so the filter is applied where the answer is read. - `partitioned_read` is `auto` by default: split the scan when the driver can, read it as one statement when it cannot. `required` forbids that downgrade, which is what keeps a test from going green while quietly exercising the fallback; `disabled` is the escape hatch. - Metadata is cached per catalog for 10 minutes by default (`meta.cache.adbc.metadata.*`), and every `REFRESH` statement drops it. A newly created remote table is reachable without any refresh. - Read only: `INSERT`, `CREATE`/`DROP TABLE` and the other write statements against an ADBC catalog are rejected. An MTMV over an ADBC table does build and refresh. ### Test coverage - `fe-connector-adbc`: 186 unit tests, no skips. About 25 of them drive the real SQLite ADBC driver through the same Java -> JNI -> C driver manager -> driver `.so` path FE takes in production; `fe-connector-api` covers the new SPI field. - BE: unit tests for the ADBC reader, the driver registry, the Arrow variant normalizer, the scanner gate and the TIMESTAMPTZ serde. - End-to-end (`regression-test/suites/external_table_p0/adbc/`): 20 suites against a live catalog -- type mapping and semantics, complex and binary types, scan edges, predicate pushdown, column pruning, query shapes, source table models, metadata operations, nested catalogs, cross-source joins, large data, outfile, MTMV, negative cases, and partitioned vs. single-statement reads compared row for row. Each suite returns early when the driver is absent, so a cluster without one does not fail. ### Known limits - Read only. No writes, DDL, statistics or aggregate pushdown. - Partitions are verified to be read completely and without duplication, but **spreading them over several backends has only been reasoned from the code**, not observed -- the test environment has one backend, and `test_adbc_multi_backend` returns early there. - `IPV4` from a Doris source arrives as a bare `INT` (both sides encode it as int32), and a source `DATETIME` arrives as `TIMESTAMPTZ`. Both are asserted, not worked around. - The FE and BE copies of the driver are not checked against each other; `driver_checksum` verifies FE's copy only. ### Release note Support ADBC catalog, which reads an external source through an Arrow Database Connectivity driver: data is transferred as Arrow record batches and a scan is split across backends by the driver's own result partitions. Phase one targets Arrow Flight SQL sources. The driver library is supplied by the operator; Doris ships none. Also fixes two defects on the Arrow Flight SQL path that are visible to existing clients: `GetTables` reported the wrong Arrow type for `DATEV2` and placeholder children for `ARRAY`/`MAP`/`STRUCT`, and a `TIMESTAMPTZ` column read from Arrow was decoded by copying its bits instead of converting the epoch.

morningman changed the title ~~[feat](catalog) Support ADBC Catalog~~[feat](catalog) support ADBC catalog that reads external sources over ArrowAug 1, 2026

morningman force-pushed the wt-adbc-catalog branch from 77ee3bd to a6e8d66Compare August 2, 2026 05:38

morningman mentioned this pull request Aug 2, 2026
[feat](thirdparty) add arrow-adbc to the thirdparty build #66358
Merged
16 tasks

morningman force-pushed the wt-adbc-catalog branch from a6e8d66 to 4315a84Compare August 2, 2026 15:51

morningman marked this pull request as ready for review August 2, 2026 15:52

morningman requested review from CalvinKirs, Gabriel39 and yiguolei as code owners August 2, 2026 15:52

morningman mentioned this pull request Aug 4, 2026
[Tracking] Multi-modal lakehouse roadmap: Iceberg, Paimon and Lance in Doris 4.2 / 5.0 #66418
Open

morningmanand others added 6 commits August 4, 2026 15:23

[feat](adbc) link adbc_driver_manager into doris_be
dc8423b
Nothing references the ADBC symbols yet, so the archive contributes no objects to the binary until AdbcDriverRegistry lands. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

morningmanand others added 7 commits August 4, 2026 15:23

format
cd8142c

morningman force-pushed the wt-adbc-catalog branch from 4315a84 to 1bc74d1Compare August 4, 2026 07:55

Gabriel39 approved these changes Aug 5, 2026
View reviewed changes

github-actionsBot added the approved Indicates a PR has been approved by one committer. label Aug 5, 2026

github-actionsBot added the reviewed label Aug 5, 2026

CalvinKirs approved these changes Aug 5, 2026
View reviewed changes

morningman merged commit 2a92450 into apache:masterAug 5, 2026
29 of 31 checks passed

CalvinKirs mentioned this pull request Aug 5, 2026
[fix](catalog) apply the jdbc driver_url checks to iceberg/paimon and to ALTER CATALOG #66483
Open
3 tasks

morningman mentioned this pull request Aug 10, 2026
[doc](lakehouse) Add ADBC Catalog documentation apache/doris-website#4050
Merged

HappenLee mentioned this pull request Aug 11, 2026
[feature](query) Generate TopN filters from aggregation #66652
Open

morningman mentioned this pull request Sep 1, 2026
[Tracking] Arrow Flight SQL and ADBC Catalog: issues and improvements #65615
Open
63 tasks

Uh oh!

Conversation

morningman commented Jul 31, 2026• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What problem does this PR solve?

Supported features

Design decisions worth knowing before reviewing

Changes outside the connector

Behaviour to be aware of

Test coverage

Known limits

Release note

Check List (For Author)

Check List (For Reviewer who merge this PR)

Uh oh!

hello-stephen commented Jul 31, 2026

Uh oh!

morningman commented Aug 2, 2026

Uh oh!

hello-stephen commented Aug 2, 2026

Cloud UT Coverage Report

Uh oh!

hello-stephen commented Aug 2, 2026

FE UT Coverage Report

Uh oh!

hello-stephen commented Aug 2, 2026

BE UT Coverage Report

Uh oh!

hello-stephen commented Aug 2, 2026

BE Regression && UT Coverage Report

Uh oh!

hello-stephen commented Aug 2, 2026

FE Regression Coverage Report

Uh oh!

morningman commented Aug 3, 2026

Uh oh!

hello-stephen commented Aug 3, 2026

FE UT Coverage Report

Uh oh!

hello-stephen commented Aug 3, 2026

Uh oh!

hello-stephen commented Aug 3, 2026

Uh oh!

hello-stephen commented Aug 3, 2026

Uh oh!

hello-stephen commented Aug 3, 2026

BE Regression && UT Coverage Report

Uh oh!

hello-stephen commented Aug 3, 2026

FE Regression Coverage Report

Uh oh!

morningman commented Aug 4, 2026

Uh oh!

hello-stephen commented Aug 4, 2026

Cloud UT Coverage Report

Uh oh!

hello-stephen commented Aug 4, 2026

FE UT Coverage Report

Uh oh!

hello-stephen commented Aug 4, 2026

Uh oh!

hello-stephen commented Aug 4, 2026

Uh oh!

hello-stephen commented Aug 4, 2026

Uh oh!

hello-stephen commented Aug 4, 2026

FE Regression Coverage Report

Uh oh!

morningman commented Aug 4, 2026

Uh oh!

github-actionsBot commented Aug 5, 2026

Uh oh!

github-actionsBot commented Aug 5, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

morningman commented Jul 31, 2026•
edited
Loading