feat(rust/core): add async traits for Driver, Database, Connection, Statement by if0ne · Pull Request #3714 · apache/arrow-adbc

if0ne · 2025-11-14T08:07:33Z

Hi, I’m back again. This is a continuation of this PR: #3712

I’ve returned with a draft of the asynchronous traits. They fully mirror the synchronous API (with a couple of differences in input arguments), adding Future + Send as the return type.

As further development, I can propose:

Abstract away from async executors and introduce an AsyncExecutor trait with a single block_on method. In adbc_core, add helper structures like SyncDriverWrapper<E: AsyncExecutor, D: AsyncDriver> and others, and implement the corresponding synchronous traits for them.
Add some procedural macro magic to automatically implement the synchronous traits by generating the code described above.

if0ne · 2025-11-14T08:08:24Z

What do you think? @lidavidm and @eitsupi

lidavidm · 2025-11-14T08:16:36Z

CC @felipecrv @mbrobbel too

mbrobbel

For the Send bound stuff I would suggest using https://docs.rs/trait-variant/0.1.2/trait_variant/ (as suggested in https://blog.rust-lang.org/2023/12/21/async-fn-rpit-in-traits/#async-fn-in-public-traits).

For async vs sync I would suggest using the async trait as the default and provided sync helper traits and structs. But the truth is; there is no elegant solution available today for this.

if0ne · 2025-11-14T11:54:37Z

trait_variant will help solve the Send bound problem, but it introduces another problem: the macro does not add additional Send bound to function input arguments if they are of an opaque type. Should we use explicit types instead?

felipecrv

Once the code in blocking is moved back to the current location I can give another review.

@lidavidm do we really need everything to be a Future, even get/set options?

if0ne · 2025-11-14T18:40:47Z

Once the code in blocking is moved back to the current location I can give another review.
@lidavidm do we really need everything to be a Future, even get/set options?

I made the AsyncOptionable because in the DataFusion implementation, setting options involved asynchronous operations. It might be possible to make get_option_* sync, but I would keep set_option async.

https://github.com/if0ne/arrow-adbc/blob/61963f8e450926da1657f31112db9498f789b380/rust/driver/datafusion/src/lib.rs#L247

lidavidm · 2025-11-16T00:08:46Z

Even get_option may involve I/O in some cases (e.g. if the driver has to run a query to fetch some information), unfortunately

lidavidm · 2025-12-05T06:20:47Z

Does arrow-rs not already have a trait for this?

I see that arrow-rs does not implement ArrowAsyncDeviceStreamHandler etc yet. I think we need to figure this out in arrow-rs

Related issue apache/arrow-rs#7228

Filed apache/arrow-rs#9166

After the merge, I’ll get back to this PR

Merged (at least the trait). We can bump to the next version when available (possibly via feature? Or maybe splitting async traits into a new crate?)

I will split the current PR into three pull requests:
Moving the existing structures into a separate module
Adding sendable async traits as an async feature
Adding bridge structures, which will probably also be under the async feature

lidavidm · 2025-12-05T06:29:37Z

I think there needs to be explanation somewhere of what "local" means (traits that are not Send?)
Additionally, does it make sense to have non-Send variants? It's not clear to me that that's useful (when would a connection be limited to a single thread?)

This is about flexibility. I think that, in practical terms, most drivers will implement the async trait with a Send bound. But someone might want to implement an async Rust driver that uses an FFI library to connect to a specialized database.

Is it actually worth optimizing for that flexibility, though? (Versus, say, expecting that such a very specialized database would instead manage this internally?)

Because as it stands, this means duplicating every method/trait

IMO it’s not worth supporting the non-Send variant (maybe someday in the future, when the crate stabilizes and reaches versions > 1.0.0).
Personally, I was planning to use adbc for the wasm target, and as far as I understand, all the necessary structures are Send.

FYI: I found another driver (which seems to be fairly recent): https://github.com/marconae/exarrow-rs

The sedona-db implementation is unused to my knowledge so don't worry about any compatibility issues there!

@marconae sorry for the random ping, but would you like to weigh in on async ADBC interfaces? 🙂

This is about flexibility.
We might end up offering too much flexibility in a standard that is supposed to make database access more uniform. If a database doesn't allow its handles to move between threads, it's the responsibility of the ADBC driver wrapping these database APIs to fix for that rather than bubbling up these restrictions to Rust users of ADBC via the type system which basically means a rewrite of the entire application depending on the ADBC driver that it uses.

@lidavidm thx for including me. I had to go async, because I am fetching data from an async web-socket protocol. Maybe the my pattern is a bit unusual. For me it was fine to work against a synchronous library and wrap it.

…or DataFusion driver Signed-off-by: if0ne <pavel.agafonov.al@gmail.com>

Signed-off-by: if0ne <pavel.agafonov.al@gmail.com>

…d sync wrappers Signed-off-by: if0ne <pavel.agafonov.al@gmail.com>

Signed-off-by: if0ne <pavel.agafonov.al@gmail.com>

A small step toward implementing #3714 Signed-off-by: if0ne <pavel.agafonov.al@gmail.com>

Adds a short §20 to the design covering the convergence plan with apache/arrow-adbc#3714 (active upstream PR that adds async traits to adbc_core). Three items: 1. Adopt RPITIT style (`fn foo() -> impl Future<Output=T> + Send`) instead of `#[async_trait]` from day one. Supersedes prior references to async-trait in §5.4/§5.5. Zero-cost convergence when #3714 lands. 2. Convergence is a mechanical ~200 LOC refactor when upstream merges — no behavioral change, napi-rs binding unaffected. Our AsyncResultReader becomes Pin<Box<dyn RecordBatchStream + Send>>; our inherent methods become trait impls; our Databricks-specific metadata method variants go away since upstream `get_objects` covers them. Our AsyncAuthProvider stays — ADBC has no auth trait. 3. Deep-dive analysis of the upstream PR and the Apache Node driver manager (@apache-arrow/adbc-driver-manager) lives in a companion reference doc (adbc-upstream-references.md) to keep the design focused on architecture rather than investigation notes. Why this was not folded into §19 (Apache Node binding validation): different subject matter (runtime trait shape vs binding-layer patterns), different convergence paths (we already implement adbc_core sync traits; we'll add async ones; we don't adopt the Node package). Co-authored-by: Isaac

if0ne requested a review from wjones127 as a code owner November 14, 2025 08:07

if0ne marked this pull request as draft November 14, 2025 08:07

github-actionsBot added this to the ADBC Libraries 22 milestone Nov 14, 2025

if0ne force-pushed the rust/async-trait branch from a52ee39 to ad1f184Compare November 14, 2025 08:18

if0ne changed the title ~~feat(rust/adbc_core): add async traits for Driver, Database, Connection, Statement~~feat(rust/core): add async traits for Driver, Database, Connection, StatementNov 14, 2025

mbrobbel reviewed Nov 14, 2025
View reviewed changes

if0ne force-pushed the rust/async-trait branch 2 times, most recently from 2888512 to 61963f8Compare November 14, 2025 14:09

felipecrv reviewed Nov 14, 2025
View reviewed changes

Comment threadrust/core/src/non_blocking.rs Outdated
Comment threadrust/core/src/non_blocking.rs Outdated
Comment threadrust/core/src/non_blocking.rs Outdated
Comment threadrust/core/src/sync.rs

if0ne force-pushed the rust/async-trait branch from 61963f8 to c838f22Compare November 26, 2025 08:11

if0ne marked this pull request as ready for review November 26, 2025 10:25

lidavidm reviewed Dec 5, 2025
View reviewed changes

if0ne added 13 commits December 5, 2025 13:09

add: async traits for driver, db, connection, stmt and implement it f…
0a09291
…or DataFusion driver Signed-off-by: if0ne <pavel.agafonov.al@gmail.com>

add: license for new files
ce0a380
Signed-off-by: if0ne <pavel.agafonov.al@gmail.com>

fix: docs for async traits
91a59bc
Signed-off-by: if0ne <pavel.agafonov.al@gmail.com>

add: trait-variant for Send bound
4389ece
Signed-off-by: if0ne <pavel.agafonov.al@gmail.com>

changes: re-export non_blocking mod, blocking mod for sync traits, ad…
a8ce311
…d sync wrappers Signed-off-by: if0ne <pavel.agafonov.al@gmail.com>

fix: docs
4ed60ba
Signed-off-by: if0ne <pavel.agafonov.al@gmail.com>

changes: remove trait-variant deps
5f7e867
Signed-off-by: if0ne <pavel.agafonov.al@gmail.com>

changes: renamed blocking into sync
438b1c0
Signed-off-by: if0ne <pavel.agafonov.al@gmail.com>

feat: add RecordBatchStream trait for async traits
4852851
Signed-off-by: if0ne <pavel.agafonov.al@gmail.com>

fix: fmt
6f93a88
Signed-off-by: if0ne <pavel.agafonov.al@gmail.com>

fix: docs
9706b6f
Signed-off-by: if0ne <pavel.agafonov.al@gmail.com>

changes: sync traits by default
6da2a96
Signed-off-by: if0ne <pavel.agafonov.al@gmail.com>

fix: docs
f0c6ae9
Signed-off-by: if0ne <pavel.agafonov.al@gmail.com>

changes: moved all sync wrappers into syncify module
4996555
Signed-off-by: if0ne <pavel.agafonov.al@gmail.com>

if0ne force-pushed the rust/async-trait branch from 65929ed to 4996555Compare December 5, 2025 10:25

lidavidm modified the milestones: ADBC Libraries 22, ADBC Libraries 23 Jan 6, 2026

if0ne mentioned this pull request Jan 30, 2026
refactor(rust/core): moved adbc traits to sync mod #3932
Merged

lidavidm pushed a commit that referenced this pull request Feb 9, 2026
refactor(rust/core): moved adbc traits to sync mod (#3932)
7cc8b6a
A small step toward implementing #3714 Signed-off-by: if0ne <pavel.agafonov.al@gmail.com>

if0ne mentioned this pull request Feb 24, 2026
feat(rust/core): add async traits #4014
Open

lidavidm removed this from the ADBC Libraries 23 milestone Apr 7, 2026

lidavidm mentioned this pull request Apr 23, 2026
docs(rust): add async public API design adbc-drivers/databricks#398
Closed

Uh oh!

Conversation

if0ne commented Nov 14, 2025

Uh oh!

if0ne commented Nov 14, 2025

Uh oh!

lidavidm commented Nov 14, 2025

Uh oh!

mbrobbel left a comment

Choose a reason for hiding this comment

Uh oh!

if0ne commented Nov 14, 2025

Uh oh!

felipecrv left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

if0ne commented Nov 14, 2025

Uh oh!

lidavidm commented Nov 16, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

marconaeJan 17, 2026• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

marconaeJan 17, 2026•
edited
Loading