`select array_union([[null]], [])` failing as not all bytes were consumed in `RowConverter`

[Jefffrey]

Describe the bug

Running locally:

datafusion (main)$ cargo test --test sqllogictests array
 Finished `test` profile [unoptimized + debuginfo] target(s) in 0.61s
 Running bin/sqllogictests.rs (/Users/jeffrey/.cargo_target_cache/debug/deps/sqllogictests-a21d0d1e87978561)
Completed 7 test files in 2 seconds 
External error: 1 errors in file /Users/jeffrey/Code/datafusion/datafusion/sqllogictest/test_files/array.slt

1. query failed: DataFusion error: Arrow error: Invalid argument error: Codecs [List(RowConverter { fields: [SortField { options: SortOptions { descending: false, nulls_first: true }, data_type: Null }], codecs: [Stateless] })] did not consume all bytes for row 0, remaining bytes: [1]
[SQL] select array_union([[null]], []);
at /Users/jeffrey/Code/datafusion/datafusion/sqllogictest/test_files/array.slt:4750



Error: Execution("1 failures")
error: test failed, to rerun pass `-p datafusion-sqllogictest --test sqllogictests`

Caused by:
 process didn't exit successfully: `/Users/jeffrey/.cargo_target_cache/debug/deps/sqllogictests-a21d0d1e87978561 array` (exit status: 1)

Related code in arrow-rs:

https://github.com/apache/arrow-rs/blob/f122d7786837d1fc6d93ad45c13067fbfe93eb3e/arrow-row/src/lib.rs#L936-L945

To Reproduce

No response

Expected behavior

No response

Additional context

No response

[kumarUjjawal]

Should the output be [[NULL]] (preserving the null element) or [[]] (treating null as empty)?

[Jefffrey]

Should the output be [[NULL]] (preserving the null element) or [[]] (treating null as empty)?

This test is an existing one so we should follow what the current (expected) output is

[kumarUjjawal]

Thanks @Jefffrey

[kumarUjjawal]

take