Skip to content

rewrite array_append/array_prepend to remove deplicate codes - #8108

Merged
alamb merged 3 commits into
apache:mainfrom
Veeupup:array_append
Nov 10, 2023
Merged

rewrite array_append/array_prepend to remove deplicate codes #8108
alamb merged 3 commits into
apache:mainfrom
Veeupup:array_append

Conversation

@Veeupup

@VeeupupVeeupup commented Nov 9, 2023

Copy link
Copy Markdown
Contributor

Which issue does this PR close?

One issue from #7988array_append && array_prepend.

Rationale for this change

What changes are included in this PR?

Are these changes tested?

Are there any user-facing changes?

@github-actionsgithub-actionsBot added physical-expr Changes to the physical-expr crates core Core DataFusion crate labels Nov 9, 2023
@Veeupup

Veeupup commented Nov 9, 2023

Copy link
Copy Markdown
ContributorAuthor

@alamb@jayzhan211 PTAL ^ . ^

Signed-off-by: veeupup <code@tanweime.com>
@VeeupupVeeupup changed the title rewrite array_append to remove deplicate codesrewrite array_append/array_prepend to remove deplicate codes Nov 9, 2023
@alamb

alamb commented Nov 9, 2023

Copy link
Copy Markdown
Contributor

Thanks @Veeupup -- I'll check it out shortly

@Weijun-HWeijun-H left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you @Veeupup ! Could you also move the related tests to sqllogictest?

@jayzhan211

Copy link
Copy Markdown
Contributor

My expected way for array_append/array_prepend is building array via MutableArrayData, so I need to think about whether row converter is the better choice or not.

@jayzhan211

Copy link
Copy Markdown
Contributor

I wrote another function with MutableArrayData and it is a lot faster. 30000 micro second vs 300 micro second. @Veeupup Can you also try with MutableArrayData and compare it yourself, if you also got the same result, we can go with MutableArrayData approach.

@Veeupup

Copy link
Copy Markdown
ContributorAuthor

Thank you @Veeupup ! Could you also move the related tests to sqllogictest?

Sure! I'll make it later

@Veeupup

Copy link
Copy Markdown
ContributorAuthor

I wrote another function with MutableArrayData and it is a lot faster. 30000 micro second vs 300 micro second. @Veeupup Can you also try with MutableArrayData and compare it yourself, if you also got the same result, we can go with MutableArrayData approach.

Appreciate your advice! I'll try this way and benchmark it then

@Veeupup

Copy link
Copy Markdown
ContributorAuthor

Thank you @Veeupup ! Could you also move the related tests to sqllogictest?

hi @Weijun-H ! it seems that I do not need to do anything? (Or just remove ut fn test_array_prepend and fn test_array_append)

I have checked the sqllogictests with array_append/prepend, and it seems that sqllogictests already have these cases.

Signed-off-by: veeupup <code@tanweime.com>
Signed-off-by: veeupup <code@tanweime.com>
@Veeupup

Copy link
Copy Markdown
ContributorAuthor

@jayzhan211 I have implemented the array_append in MutableArray and RowConverter and it shows that Mutable performs better too.

Here are the benchmarks for sqllogictests:

MutableArrayRowConverter
29.297µs93.433µs
20.555µs32.201µs
19.251µs86.561µs
102.495µs143.772µs
58.49µs35.456µs
56.035µs33.316µs
45.607µs79.701µs
23.508µs33.592µs
22.87µs37.584µs
39.27µs56.071µs
23.973µs35.368µs
24.97µs37.753µs
44.644µs59.515µs
26.602µs35.498µs
22.272µs37.957µs
97.846µs169.643µs
107.068µs123.551µs
55.058µs178.546µs
45.853µs109.962µs
82.722µs90.556µs
43.62µs44.157µs

@alamb

Copy link
Copy Markdown
Contributor

Very nice!

@alambalamb left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you @Veeupup and @jayzhan211 -- this looks a lot nicer to me. I think there is even more performance to be gained here by avoiding concat (and using ArrayData to build up the final output) but this is a lot better than what we have currently.

Thanks a lot 🙏

Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

coreCore DataFusion cratephysical-exprChanges to the physical-expr crates

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants

@Veeupup@alamb@jayzhan211@Weijun-H