Skip to content

Early exit on column normalisation to improve DataFrame performance - #14636

Merged
alamb merged 3 commits into
apache:mainfrom
blaginin:early-exit-on-normalization
Feb 17, 2025
Merged

Early exit on column normalisation to improve DataFrame performance#14636
alamb merged 3 commits into
apache:mainfrom
blaginin:early-exit-on-normalization

Conversation

@blaginin

Copy link
Copy Markdown
Member

Which issue does this PR close?

Related to #14563 (probably more prs to come)

Rationale for this change

Now, when normalizing the column, we always generate plan.using_columns() which is recursive and very expensive - and may not be needed if column is already normalized

What changes are included in this PR?

Exit early if column already has a relation set. Also, set the relation when with_column_renamed is called

Are these changes tested?

Extended a test to assert references

Are there any user-facing changes?

No

@github-actionsgithub-actionsBot added logical-expr Logical plan and expressions core Core DataFusion crate labels Feb 12, 2025
@blaginin

Copy link
Copy Markdown
MemberAuthor

Got +38% increase in dataframe benchmark

 before after
with_column_10 769.06 µs 673.32 µs
with_column_100 1.4952 s 978.43 ms
with_column_200 36.544 s 26.682 s

@alamb

Copy link
Copy Markdown
Contributor

FYI @Omega359

@Omega359

Copy link
Copy Markdown
Contributor

I'll check this out tomorrow. we've been chatting about our approaches on #14563

@Omega359

Copy link
Copy Markdown
Contributor

I think this is a reasonable change. I think it could be incorporated into my upcoming PR to enhance things a bit, especially since I think it may help other dataframe functions such as select(exprs)

@blaginin
blaginin marked this pull request as ready for review February 13, 2025 15:40
@blaginin

Copy link
Copy Markdown
MemberAuthor

Nice thank you! But let's maybe keep PRs atomic? I plan to do one more (the one I described as a third in the issue), I don't think they overlap with each other?

@Omega359

Copy link
Copy Markdown
Contributor

Of course we can have them atomic :)

Comment threaddatafusion/core/src/dataframe/mod.rs
Comment on lines +837 to +842
let column = column.into();
if column.relation.is_some() {
// column is already normalized
return Ok(column);
}

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks like a no brainer performance boost. Thanks!

@alambalamb changed the title Early exit on column normalisationEarly exit on column normalisation to improve DataFrame performanceFeb 14, 2025

@alambalamb left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks like a no brainer performance boost. Thanks!

I agree -- thank you @blaginin@timsaucer and @Omega359

🚀

@alamb
alamb merged commit 580e622 into apache:mainFeb 17, 2025
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

coreCore DataFusion cratelogical-exprLogical plan and expressions

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants

@blaginin@alamb@Omega359@timsaucer