Fix AvroReader: Add union resolving for nested struct arrays

[JonasDev1]

Which issue does this PR close?

Resolves#12682

What changes are included in this PR?

Nullable values are passed as unions and this requires a resolving of the union to have the correct null_buffer size.

Are these changes tested?

Added a new test for this issue

[alamb]

Thank you for this contribution @JonasDev1 -- I started the CI and I left a comment.

[alamb]

I ran this test without the changes in this PR and it still passes 🤔 Thus I don't think it covers the code change

andrewlamb@Andrews-MacBook-Pro-2:~/Software/datafusion$ git diff
diff --git a/datafusion/core/src/datasource/avro_to_arrow/arrow_array_reader.rs b/datafusion/core/src/datasource/avro_to_arrow/arrow_array_reader.rs
index 24bc4a61c..8735eac5f 100644
--- a/datafusion/core/src/datasource/avro_to_arrow/arrow_array_reader.rs+++ b/datafusion/core/src/datasource/avro_to_arrow/arrow_array_reader.rs@@ -573,7 +573,7 @@ impl<'a, R: Read> AvroArrowArrayReader<'a, R> {
// extract list values, with non-lists converted to Value::Null
let array_item_count = rows
.iter()
- .map(|row| match maybe_resolve_union(row) {+ .map(|row| match row {
Value::Array(values) => values.len(),
_ => 1,
})

And then I ran

andrewlamb@Andrews-MacBook-Pro-2:~/Software/datafusion$ cargo test --lib -p datafusion --all-features -- avro_to_arrow
Finished `test` profile [unoptimized + debuginfo] target(s) in 0.12s
Running unittests src/lib.rs (target/debug/deps/datafusion-3bc79b97e7a0ad1b)
running 16 tests
test datasource::avro_to_arrow::schema::test::test_invalid_avro_schema ... ok
test datasource::avro_to_arrow::schema::test::test_non_record_schema ... ok
test datasource::avro_to_arrow::schema::test::test_alias ... ok
test datasource::avro_to_arrow::schema::test::test_plain_types_schema ... ok
test datasource::avro_to_arrow::schema::test::test_external_props ... ok
test datasource::avro_to_arrow::schema::test::test_nested_schema ... ok
test datasource::avro_to_arrow::arrow_array_reader::test::test_avro_read_nested_list ... ok
test datasource::avro_to_arrow::arrow_array_reader::test::test_time_avro_milliseconds ... ok
test datasource::avro_to_arrow::reader::tests::test_avro_basic ... ok
test datasource::avro_to_arrow::arrow_array_reader::test::test_avro_read_list ... ok
test datasource::avro_to_arrow::arrow_array_reader::test::test_avro_iterator ... ok
test datasource::avro_to_arrow::arrow_array_reader::test::test_avro_nullable_struct ... ok
test datasource::avro_to_arrow::arrow_array_reader::test::test_avro_nullable_struct_array ... ok
test datasource::avro_to_arrow::arrow_array_reader::test::test_complex_list ... ok
test datasource::avro_to_arrow::arrow_array_reader::test::test_deep_nullable_struct ... ok
test datasource::avro_to_arrow::arrow_array_reader::test::test_complex_struct ... ok
test result: ok. 16 passed; 0 failed; 0 ignored; 0 measured; 729 filtered out; finished in 0.00s
andrewlamb@Andrews-MacBook-Pro-2:~/Software/datafusion$

[JonasDev1]

Thank you for the comment. Yes the test rather checks the happy path. I will add a test for the index error tomorrow and update the formating.

[JonasDev1]

I updated the test. It now fails without the change:

thread 'datasource::avro_to_arrow::arrow_array_reader::test::test_avro_nullable_struct_array' panicked at /Users/JONSchmi/.cargo/registry/src/index.crates.io-6f17d22bba15001f/arrow-buffer-52.2.0/src/util/bit_util.rs:55:5:
index out of bounds: the len is 1 but the index is 1

[alamb]

Indeed -- I double checked and it does fail!

[alamb]

Thank you very much @JonasDev1

This looks great to me.

[alamb]

Since I had this PR checkout anyways to test it, I also took the liberty to push a commit that merges with main and fixes cargo fmt CI failure

[alamb]

CI failure is due to a network problem: https://github.com/apache/datafusion/actions/runs/11142914457/job/30967142581?pr=12686

3s
Prepare all required actions
Getting action download info
Run ./.github/actions/setup-builder
Run apt-get update
Get:[1](https://github.com/apache/datafusion/actions/runs/11142914457/job/30967142581?pr=12686#step:6:1) http://deb.debian.org/debian bullseye InRelease [116 kB]
Get:[2](https://github.com/apache/datafusion/actions/runs/11142914457/job/30967142581?pr=12686#step:6:2) http://deb.debian.org/debian-security bullseye-security InRelease [27.2 kB]
Get:[3](https://github.com/apache/datafusion/actions/runs/11142914457/job/30967142581?pr=12686#step:6:3) http://deb.debian.org/debian bullseye-updates InRelease [44.1 kB]
Get:4 http://deb.debian.org/debian bullseye/main amd64 Packages [8066 kB]
Get:5 http://deb.debian.org/debian-security bullseye-security/main amd64 Packages [296 kB]
Get:6 http://deb.debian.org/debian bullseye-updates/main amd64 Packages [18.8 kB]
Fetched 8568 kB in 1s (7458 kB/s)
Reading package lists...
Reading package lists...
Building dependency tree...
Reading state information...
The following additional packages will be installed:
libprotobuf-dev libprotobuf-lite23 libprotobuf23 libprotoc23
Suggested packages:
protobuf-mode-el
The following NEW packages will be installed:
libprotobuf-dev libprotobuf-lite23 libprotobuf23 libprotoc23
protobuf-compiler
0 upgraded, 5 newly installed, 0 to remove and 3 not upgraded.
Need to get 3245 kB of archives.
After this operation, 19.1 MB of additional disk space will be used.
Get:1 http://deb.debian.org/debian bullseye/main amd64 libprotobuf23 amd64 3.12.4-1+deb11u1 [891 kB]
Err:1 http://deb.debian.org/debian bullseye/main amd64 libprotobuf23 amd64 3.12.4-1+deb11u1
Error reading from server - read (104: Connection reset by peer) [IP: 199.232.98.132 80]
Get:2 http://deb.debian.org/debian bullseye/main amd64 libprotobuf-lite23 amd64 3.12.4-1+deb11u1 [242 kB]
Get:3 http://deb.debian.org/debian bullseye/main amd64 libprotobuf-dev amd64 3.12.4-1+deb11u1 [1234 kB]
Get:4 http://deb.debian.org/debian bullseye/main amd64 libprotoc23 amd64 3.12.4-1+deb11u1 [802 kB]
Get:5 http://deb.debian.org/debian bullseye/main amd64 protobuf-compiler amd64 3.12.4-1+deb11u1 [75.3 kB]
E: Failed to fetch http://deb.debian.org/debian/pool/main/p/protobuf/libprotobuf23_3.12.4-1%2bdeb11u1_amd64.deb Error reading from server - read (104: Connection reset by peer) [IP: 199.2[32](https://github.com/apache/datafusion/actions/runs/11142914457/job/30967142581?pr=12686#step:6:34).98.132 80]
Fetched 2354 kB in 0s (8955 kB/s)
E: Unable to fetch some archives, maybe run apt-get update or try with --fix-missing?
Error: Process completed with exit code 100.

Which is not related to this PR

[alamb]

Thanks again @JonasDev1