Skip to content

Implementation of array_intersect - #8081

Merged
xudong963 merged 2 commits into
apache:mainfrom
Veeupup:array_interact
Nov 11, 2023
Merged

Implementation of array_intersect#8081
xudong963 merged 2 commits into
apache:mainfrom
Veeupup:array_interact

Conversation

@Veeupup

@VeeupupVeeupup commented Nov 7, 2023•
edited
Loading

Copy link
Copy Markdown
Contributor

Which issue does this PR close?

Closes#6978

Rationale for this change

Using RowConverter to implement array_intersect, no duplicate code for different data types,

What changes are included in this PR?

Are these changes tested?

Are there any user-facing changes?

@github-actionsgithub-actionsBot added logical-expr Logical plan and expressions physical-expr Changes to the physical-expr crates core Core DataFusion crate labels Nov 7, 2023
@Veeupup

Veeupup commented Nov 7, 2023•
edited
Loading

Copy link
Copy Markdown
ContributorAuthor

hi @edmondop@jayzhan211 , I have a draft implementation for array_intersect, I use a hashmap to store the first array values and find values that appear in the hashmap. And it works just fine and looks like a general approach?

But I encounter a problem for Float32 and Float64, we can not just use HashMap<f32/f64> because std::hash::Hash is not implemented for float types, so I want to fix it by introducing ordered-float to make it hashable, is it too heavy for this case ?
Or do you have any better ideas to make it？ Very happy to see your advice ！

@tustvold

tustvold commented Nov 7, 2023•
edited
Loading

Copy link
Copy Markdown
Contributor

Perhaps you could use https://docs.rs/arrow-row/latest/arrow_row/? This would have the added benefit of supporting more complex types, e.g. lists of lists, and not adding additional codegen (#7988). The notes I wrote on #6981 (comment) might be helpful for this

tustvold
tustvold reviewed Nov 8, 2023
View reviewed changes
Comment threaddatafusion/physical-expr/src/array_expressions.rs Outdated
Comment threaddatafusion/physical-expr/src/array_expressions.rs Outdated
@VeeupupVeeupup mentioned this pull request Nov 8, 2023
Closed
@VeeupupVeeupup changed the title Initial Implementation of array_intersectImplementation of array_intersectNov 8, 2023
@Veeupup
Veeupup marked this pull request as ready for review November 8, 2023 16:38
Dandandan
Dandandan reviewed Nov 9, 2023
View reviewed changes
Comment threaddatafusion/physical-expr/src/array_expressions.rs Outdated
@github-actionsgithub-actionsBot added the sqllogictest SQL Logic Tests (.slt) label Nov 9, 2023
jayzhan211
jayzhan211 requested changes Nov 9, 2023
View reviewed changes
Comment threaddatafusion/expr/src/built_in_function.rs Outdated
Comment threaddatafusion/physical-expr/src/array_expressions.rs Outdated
Comment threaddatafusion/physical-expr/src/array_expressions.rs Outdated
@Veeupup
Veeupupforce-pushed the array_interact branch 2 times, most recently from 6b7198f to f12525cCompareNovember 9, 2023 12:42
Veeupup
Veeupup commented Nov 9, 2023
View reviewed changes
Comment threaddatafusion/core/src/datasource/avro_to_arrow/arrow_array_reader.rs
jayzhan211
jayzhan211 reviewed Nov 9, 2023
View reviewed changes
Comment threaddatafusion/physical-expr/src/array_expressions.rs Outdated
jayzhan211
jayzhan211 reviewed Nov 9, 2023
View reviewed changes
Comment threaddatafusion/physical-expr/src/array_expressions.rs Outdated
jayzhan211
jayzhan211 reviewed Nov 9, 2023
View reviewed changes
Comment threaddatafusion/physical-expr/src/array_expressions.rs Outdated
jayzhan211
jayzhan211 approved these changes Nov 9, 2023
View reviewed changes

@jayzhan211jayzhan211 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Roughly LGTM

@Veeupup

Veeupup commented Nov 9, 2023

Copy link
Copy Markdown
ContributorAuthor

@alamb PTAL : )

@xudong963

xudong963 commented Nov 9, 2023

Copy link
Copy Markdown
Member

@Veeupup Triggered the ci, please fix it.

@Veeupup
Initial Implementation of array_intersect
9780b5f
Signed-off-by: veeupup <code@tanweime.com>
@Veeupup
fix comments
d55fa60
Signed-off-by: veeupup <code@tanweime.com>
x
@Veeupup

Veeupup commented Nov 10, 2023•
edited
Loading

Copy link
Copy Markdown
ContributorAuthor

@xudong963 hi try fixing ci locally, maybe you can help me trigger it again or review it? : )

alamb
alamb approved these changes Nov 10, 2023
View reviewed changes

@alambalamb left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks like a nice implementation to me -- thank you @Veeupup@Dandandan@tustvold@xudong963 and @Dandandan -- what a team effort ❤️

xudong963
xudong963 approved these changes Nov 11, 2023
View reviewed changes

@xudong963xudong963 left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, welcome to arrow-datafusion

@xudong963
xudong963 merged commit 8966dc0 into apache:mainNov 11, 2023
@haohuaijinhaohuaijin mentioned this pull request Nov 11, 2023
Merged
@jayzhan211jayzhan211 mentioned this pull request Nov 11, 2023
Merged
@Veeupup
Veeupup deleted the array_interact branch November 21, 2023 15:07
@matthewgappmatthewgapp mentioned this pull request Jan 11, 2024
Closed
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@alambalambalamb approved these changes
@jayzhan211jayzhan211jayzhan211 approved these changes
@xudong963xudong963xudong963 approved these changes
@DandandanDandandanAwaiting requested review from Dandandan
@tustvoldtustvoldAwaiting requested review from tustvold

Assignees

No one assigned

Labels

coreCore DataFusion cratelogical-exprLogical plan and expressionsphysical-exprChanges to the physical-expr cratessqllogictestSQL Logic Tests (.slt)

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Implement array_intersect function

6 participants

@Veeupup@tustvold@xudong963@Dandandan@alamb@jayzhan211