Skip to content

make array_union/array_except/array_intersect handle empty/null arrays rightly - #8269

Merged
alamb merged 5 commits into
apache:mainfrom
Veeupup:fix_empty_for_array_set
Nov 21, 2023
Merged

make array_union/array_except/array_intersect handle empty/null arrays rightly#8269
alamb merged 5 commits into
apache:mainfrom
Veeupup:fix_empty_for_array_set

Conversation

@Veeupup

@VeeupupVeeupup commented Nov 19, 2023

Copy link
Copy Markdown
Contributor

Which issue does this PR close?

Closes#8181.

for array_union:

array_union([], []) = []
array_union([], null) = []
array_union(null, []) = []
array_union(null, null) = null

for array_except:

array_except([], []) = []
array_except([], null) = []
array_except(null, []) = null
array_except(null, null) = null

for array_intersect:

array_intersect([], []) = []
array_intersect([], null) = []
array_intersect(null, []) = []
array_intersect(null, null) = null

Rationale for this change

What changes are included in this PR?

Are these changes tested?

Are there any user-facing changes?

Signed-off-by: veeupup <code@tanweime.com>
@github-actionsgithub-actionsBot added logical-expr Logical plan and expressions physical-expr Changes to the physical-expr crates sqllogictest SQL Logic Tests (.slt) labels Nov 19, 2023
@Veeupup

Veeupup commented Nov 19, 2023

Copy link
Copy Markdown
ContributorAuthor

@jayzhan211 maybe you can check the behavior for these functions now cc @alamb

Signed-off-by: veeupup <code@tanweime.com>
@VeeupupVeeupup changed the title make array_union handle empty/null arrays rightlymake array_union/array_except/array_intersect handle empty/null arrays rightlyNov 19, 2023
@Veeupup
Veeupup marked this pull request as ready for review November 19, 2023 15:06
Signed-off-by: veeupup <code@tanweime.com>
@Veeupup
Veeupupforce-pushed the fix_empty_for_array_set branch from 9ed1fca to c08d6cbCompareNovember 19, 2023 15:10

@jayzhan211jayzhan211 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we need to fix here.

https://github.com/apache/arrow-datafusion/blob/393e48f98872c696a90fce033fa584533d2326fa/datafusion/sql/src/expr/value.rs#L156-L158

if data_types.is_empty(){// ListArray[NullArray(0)], the same as `make_array()`.let array = new_null_array(&DataType::Null,0);let array = Arc::new(array_into_list_array(array));Ok(lit(ScalarValue::List(array)))}elseif data_types.len() > 1{

Comment threaddatafusion/physical-expr/src/array_expressions.rs Outdated
Comment threaddatafusion/expr/src/built_in_function.rs Outdated
@jayzhan211

Copy link
Copy Markdown
Contributor

I think we need to fix here.

https://github.com/apache/arrow-datafusion/blob/393e48f98872c696a90fce033fa584533d2326fa/datafusion/sql/src/expr/value.rs#L156-L158

if data_types.is_empty(){// ListArray[NullArray(0)], the same as `make_array()`.let array = new_null_array(&DataType::Null,0);let array = Arc::new(array_into_list_array(array));Ok(lit(ScalarValue::List(array)))}elseif data_types.len() > 1{

Update: I found we can easily reuse MakeArray for sql_array_literal

Ok(Expr::ScalarFunction(ScalarFunction::new(BuiltinScalarFunction::MakeArray,
values,)))

@Veeupup

Copy link
Copy Markdown
ContributorAuthor

@jayzhan211 Thanks for your constructive comments!I suppose we make it in a right way : )

Signed-off-by: veeupup <code@tanweime.com>
@Veeupup
Veeupupforce-pushed the fix_empty_for_array_set branch from a76beaa to be32301CompareNovember 20, 2023 15:07
@jayzhan211jayzhan211 mentioned this pull request Nov 21, 2023
Comment threaddatafusion/sql/src/expr/value.rs Outdated
@jayzhan211

jayzhan211 commented Nov 21, 2023

Copy link
Copy Markdown
Contributor

We can move select_array_no_common_type and select_array_non_literal_type to slt

@alambalamb left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you very much @Veeupup and @jayzhan211 - this looks really nice and exactly correct 🙏

BuiltinScalarFunction::ArrayIntersect => Ok(input_expr_types[0].clone()),
BuiltinScalarFunction::ArrayUnion => Ok(input_expr_types[0].clone()),
BuiltinScalarFunction::ArrayUnion | BuiltinScalarFunction::ArrayIntersect => {
match (input_expr_types[0].clone(), input_expr_types[1].clone()) {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yes!

SELECT ARRAY_AGG([])
----
[]
[[]]

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍

@alamb
alamb merged commit 54a0247 into apache:mainNov 21, 2023
{
if !args.iter().all(|arg| {
arg.data_type().equals_datatype(data_type)
|| arg.data_type().equals_datatype(&DataType::Null)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@Veeupup Why do we need Null checking here, which function failed without this?

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

when it comes to handle multi arrays functions,their input types maybe like null, List(Int32).... Then, but we should handle null type in function body rather than return error here.

Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

logical-exprLogical plan and expressionsphysical-exprChanges to the physical-expr cratessqlSQL PlannersqllogictestSQL Logic Tests (.slt)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Support empty array for array_union, array_intersect, and array_except

3 participants

@Veeupup@jayzhan211@alamb