Migrate Avro reader to arrow-avro and remove internal conversion code by getChan · Pull Request #17861 · apache/datafusion

getChan · 2025-10-01T14:10:36Z

Which issue does this PR close?

ClosesUse arrow-avro for performance and improved type support #14097

Rationale for this change

DataFusion previously maintained custom Avro-to-Arrow conversion logic.
This PR migrates Avro reading to arrow-avro to align behavior with upstream Arrow and remove duplicated implementation.

What changes are included in this PR?

Switched DataFusion Avro reader path to arrow-avro (ReaderBuilder)
Removed internal/legacy Avro conversion paths that are no longer needed
Updated crate wiring to use arrow-avro and removed prior apache-avro dependency usage in affected paths
Updated Avro projection flow to use arrow-avro projection support
Added/updated upgrade documentation for Avro API and behavior changes

Are these changes tested?

Yes.

Added/updated Avro reader unit tests in datafusion/datasource-avro (including projection and timestamp logical types)
Updated SQL logic tests in datafusion/sqllogictest/test_files/avro.slt
Integration is covered by existing CI/test suites for affected crates

Are there any user-facing changes?

Yes.

DataFusionError::AvroError is removed.
From<apache_avro::Error> for DataFusionError is removed.
Re-export changed from datafusion::apache_avro to datafusion::arrow_avro.
Avro feature wiring changed:
- datafusion crate avro feature no longer enables datafusion-common/avro
- datafusion-proto crate avro feature no longer enables datafusion-common/avro
Avro decoding behavior now follows arrow-avro semantics, including:
- Avro string values being read as Arrow Binary in this path
- timestamp-* logical types read as UTC timezone-aware timestamps (Timestamp(..., Some("+00:00")))
- local-timestamp-* remaining timezone-naive (Timestamp(..., None))

Upgrade notes are documented in:
docs/source/library-user-guide/upgrading/53.0.0.md

alamb · 2025-10-02T10:10:04Z

❤️ amazing! Thank you @getChan
FYI @jecsand838 and @nathaniel-d-ef

alamb · 2025-10-19T20:04:52Z

Hi @getChan -- I am preparing to make an arrow release -- have you hit any blockers while integrating the new arrow-avro crate into DataFusion?

getChan · 2025-10-19T23:10:10Z

Hi @getChan -- I am preparing to make an arrow release -- have you hit any blockers while integrating the new arrow-avro crate into DataFusion?

No, not yet. Thanks for release.

nathaniel-d-ef · 2025-10-20T15:21:27Z

Thanks for jumping on this @getChan; let me know if I can help!

# Conflicts: # Cargo.lock # Cargo.toml # datafusion/common/Cargo.toml

alamb · 2025-10-29T11:08:03Z

FYI I merged the arrow 57 upgrade to DataFusion -- so if you rebase this PR against main you'll have access to the new arrow-avro crate

# Conflicts: # Cargo.lock

Co-authored-by: Connor Sanders <170039284+jecsand838@users.noreply.github.com>

# Conflicts: # Cargo.lock # datafusion/common/Cargo.toml

alamb

Thanks @getChan and @jecsand838 -- this is pretty epic work. The fact that all the tests pass is pretty incredible

I had a few small comments on the upgrade guide (I can help make them too if you want). Otherwise I think this is ready to go.

I also took the liberty of merging up from main to resolve a conflict

# Conflicts: # Cargo.toml

alamb · 2026-03-25T19:26:08Z

Any other thoughts @jecsand838?

FYI @Igosuki -- this is a long time since you contributed the original avro reader. It is fun to see how far things have come

jecsand838

@alamb This LGTM!

alamb · 2026-03-26T21:28:29Z

Merged up to resolve a conflict

alamb · 2026-03-26T22:14:45Z

go go go go!

alamb · 2026-03-26T22:19:11Z

Epic work @getChan

draft: using arrow-avro
e27e1f8

github-actionsBot added common Related to common crate datasource Changes to the datasource crate labels Oct 1, 2025

draft: using arrow-avro
027e79d

getChan added 3 commits October 19, 2025 17:15

Merge remote-tracking branch 'origin/main' into arrow-avro
2bf2c89

cargo fmt
2c7e364

check public API's can be replaced
7c9f976

getChan added 3 commits October 27, 2025 23:52

revert arrow dependency
7bc24ea

Merge remote-tracking branch 'origin/main' into arrow-avro
185ede5
# Conflicts: # Cargo.lock # Cargo.toml # datafusion/common/Cargo.toml

apply arrow-avro 57.0.0
51557fe

github-actionsBot removed the common Related to common crate label Oct 27, 2025

timsaucer mentioned this pull request Oct 28, 2025
Switch from deprecated xz2 to libzma #18331
Closed

getChan added 2 commits October 30, 2025 00:36

BufReader
8e12f47

remove apache-avro dependencies
83ec01d

github-actionsBot added core Core DataFusion crate common Related to common crate proto Related to proto crate labels Oct 29, 2025

getChan added 8 commits October 30, 2025 01:16

Merge remote-tracking branch 'origin/main' into arrow-avro
f37fcd3
# Conflicts: # Cargo.lock

remove apache-avro dependencies
b12f563

remove apache-avro dependencies
581ecae

fix test result
74792ff

pass projected schema to arrow-avro
151ea95

testing submodule checkout
78df065

taplo format
c7b3b63

replace with arrow-avro
2556591

getChanand others added 10 commits March 17, 2026 00:44

Apply suggestions from code review - strip mtdt
f42726d
Co-authored-by: Connor Sanders <170039284+jecsand838@users.noreply.github.com>

avro: strip metadata recursively for union types
36cf9e2

avro: preserve projection compatibility via name-based schema coercion
0019184

avro: restore missing-column test behavior to pre-17861 semantics
6831da1

avro: remap projection pushdown to writer schema ordinals
a0755b7

avro: restore compatibility docs and sqllogictest expectations
3b67bd8

Merge branch 'main' into arrow-avro
8805368
# Conflicts: # Cargo.lock # datafusion/common/Cargo.toml

avro: fix clippy in projection coercion path
43c6b6c

fix: remove unintended ahash workspace dependency changes
d5d615f

Merge branch 'main' into arrow-avro
bb0325f

alamb mentioned this pull request Mar 24, 2026
Release DataFusion 54.0.0 (Apr 2026 / May 2026) #21080
Closed
55 tasks

adriangb reviewed Mar 24, 2026
View reviewed changes

Comment threaddatafusion/datasource-avro/src/source.rs Outdated

alamb reviewed Mar 24, 2026
View reviewed changes

Comment threaddocs/source/library-user-guide/upgrading/53.0.0.md Outdated

Merge remote-tracking branch 'apache/main' into arrow-avro
b45ac7f

alamb approved these changes Mar 24, 2026
View reviewed changes

getChan added 6 commits March 25, 2026 21:19

docs: move avro upgrade note to 54 guide
d7f1d31

docs: refine avro upgrade note wording
7958154

test: remove avro sqllogictest conflict marker
da7d51a

Merge branch 'main' into arrow-avro
6c4362e
# Conflicts: # Cargo.toml

build: refresh Cargo.lock
eadd337

replace coerce_batch_to_schema to BatchAdapter
471abd3

jecsand838 approved these changes Mar 26, 2026
View reviewed changes

Merge remote-tracking branch 'apache/main' into arrow-avro
add2b3b

alamb added this pull request to the merge queue Mar 26, 2026

Merged via the queue into apache:main with commit 627fabaMar 26, 2026
34 checks passed

Uh oh!

Conversation

getChan commented Oct 1, 2025• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Which issue does this PR close?

Rationale for this change

What changes are included in this PR?

Are these changes tested?

Are there any user-facing changes?

Uh oh!

alamb commented Oct 2, 2025

Uh oh!

alamb commented Oct 19, 2025

Uh oh!

getChan commented Oct 19, 2025

Uh oh!

nathaniel-d-ef commented Oct 20, 2025

Uh oh!

alamb commented Oct 29, 2025

Uh oh!

Uh oh!

Uh oh!

alamb left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

alamb commented Mar 25, 2026

Uh oh!

jecsand838 left a comment

Choose a reason for hiding this comment

Uh oh!

alamb commented Mar 26, 2026

Uh oh!

alamb commented Mar 26, 2026

Uh oh!

Uh oh!

alamb commented Mar 26, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

getChan commented Oct 1, 2025•
edited
Loading