Convert list array and non-list array to scalars

[jayzhan211]

Which issue does this PR close?

Closes #.

Rationale for this change

convert_array_to_scalar_vec can convert well for list array, but we will get Vec<Vec<ScalarValue>> for non-list array (primitive array).
It would be nice to have another conversion that returns nicely Vec<ScalarValue> for the non-list array. I would need this for #7835 too.

What changes are included in this PR?

convert_list_array_to_scalar_vec for ListArray
convert_non_list_array_to_scalar_vec for non-ListArray

The nested array for DistinctArrayAggAccumulator (update_batch) is removed so as to test.

Are these changes tested?

Doc test

Are there any user-facing changes?

[Weijun-H]

LGTM! Thanks @jayzhan211 👍

[alamb]

Thanks @jayzhan211

[jayzhan211]

Summary of changes

1. Only accept ListArray for convert_list_array, only accept non-ListArray for convert_non_list
2. Add nested list test for convert_list_array
3. extend wrap_into_list for multiple arrays
4. keep the nested list test for distinct array agg removed, since update_batch only convert-non-list cases.

[alamb]

I think this is not waiting on any more feedback

[jayzhan211]

wait on #8253

[jayzhan211]

wait on #8439

[jayzhan211]

@alamb I think this is ready to go, thanks

[alamb]

Thank you @jayzhan211 -- this is looking quite close I think

[alamb]

👍 It might also help to explain why we need two different signatures. It took me a while to grok things too

Suggested change

	/// Convert `ListArray` to `Vec<Vec<ScalarValue>>`, first `Vec` is for rows, second `Vec` is for elements in the list
	/// Convert `ListArray` into a 2 dimensional to `Vec<Vec<ScalarValue>>`, first `Vec` is for rows,
	/// second `Vec` is for elements in the list.
	///
	/// See [`Self::convert_non_list_array_to_scalars`] for converting non Lists
	///
	/// This method is an optimization to unwrap nested ListArrays to nested Rust structures without
	/// converting them twice

[alamb]

This seems left over

Suggested change

	/// Retrieve `ScalarValue` for each row in `array`
	///

[alamb]

I think this would be easier to use if it didn't mix generic and non generic code

Maybe something like

Suggested change

	pubfn convert_list_array_to_scalar_vec<O:OffsetSizeTrait>(
	pubfn convert_list_array_to_scalar_vec(
	array:&dyn Array,
	) -> Result<Vec<Vec<Self>>> {
	ifletSome(arr) = array.as_list_opt::<i32>{
	Self::convert_list_array_to_scalar_vec_internal(arr)
	}elseifletSome(arr) = array.as_list_opt::<64>{
	Self::convert_list_array_to_scalar_vec_internal(arr)
	}else{
	_internal_err!("Expected GenericListArray but found: {array:?}")
	}
	}

And then internally pass in the cast Array by changing

 fn convert_list_array_to_scalar_vec_internal<O: OffsetSizeTrait>(
array: &dyn Array,
) -> Result<Vec<Vec<Self>>> {

to

 fn convert_list_array_to_scalar_vec_internal<O: OffsetSizeTrait>(
array: &GenericListArray<O>,
) -> Result<Vec<Vec<Self>>> {

[alamb]

(BTW sorry for being so nit picky on this PR -- I think ScalarValue is a key and important API of DataFusion so avoiding API churn (continually changing the same API over and over again) is important

[jayzhan211]

(BTW sorry for being so nit picky on this PR -- I think ScalarValue is a key and important API of DataFusion so avoiding API churn (continually changing the same API over and over again) is important

This comment is a bit too late...

And, this API is actually not breaking the previous one convert_array_to_scalar_vec, we can have three versions of them. convert_array_to_scalar_vec for the place that we can't ensure the type for array.

Anyway, this change is just improvement, not fixing a critical error, so I will close it.

[alamb]

I am sorry -- I feel really bad that we can't find more review bandwidth to match your contributions @jayzhan211 -- your efforts so far have been great and are really appreciated.