Skip to content

Merge dataframe and dataframe imp - #1998

Merged
yjshen merged 10 commits into
apache:masterfrom
vchag:merge_dataframe_and_dataframeImp
Mar 15, 2022
Merged

Merge dataframe and dataframe imp#1998
yjshen merged 10 commits into
apache:masterfrom
vchag:merge_dataframe_and_dataframeImp

Conversation

@vchag

@vchagvchag commented Mar 12, 2022•
edited
Loading

Copy link
Copy Markdown
Contributor

Which issue does this PR close?

Closes#1962 .

Rationale for this change

The Dataframe trait was introduced in order to have two separate Dataframe implementation in Datafusion and Ballista. Since then, the design has changed and we are sharing the same dataframe implementation between Datafusion and Ballista now. Therefore, we could simplify the code base by getting rid of the Dataframe trait abstraction and promoting DataframeImpl into Dataframe struct.

What changes are included in this PR?

  • Merging Dataframe trait and DataframeImpl into a single Dataframe struct
  • Deleting datafusion/src/execution/dataframe_impl.rs

venkata.chaganti added 2 commits March 12, 2022 13:56
venkata.chaganti
merging Dataframe trait and DataFrameImpl into Dataframe struct
7e4f4e9
Change-Id: Id15520a81c272a7666f17d5f3b707445a4af7798
venkata.chaganti
reformatting
f3e1139
Change-Id: I6c8cab772c206c80cd5aa5b39296d8009ab4efdb
venkata.chaganti added 3 commits March 12, 2022 14:26
venkata.chaganti
new DataFrame type fixes
3e65ef7
Change-Id: I01ebf2f52e21aaf9b24e4fb7ac54904dcd55e220
venkata.chaganti
fix build
90b3d75
Change-Id: Iefb549ecee1648a89e1165f93a8e2fa5e785df56
venkata.chaganti
re-formatting files
92d6fdf
Change-Id: I2c96a7e24f1f74052eee50ea8199f381bdb553df
xudong963
xudong963 reviewed Mar 13, 2022
View reviewed changes

@xudong963xudong963 left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, thank you @vchag

Btw, some examples in datafusion-examples/examples still use dataframe impl.

Comment threaddatafusion/src/dataframe.rs
@houqphouqp added the api change Changes the API exposed to users of the crate label Mar 13, 2022
@vchag

vchag commented Mar 14, 2022

Copy link
Copy Markdown
ContributorAuthor

LGTM, thank you @vchag

Btw, some examples in datafusion-examples/examples still use dataframe impl.

You're welcome @xudong963
Thanks for pointing it out.

The log mentions of an
unresolved import datafusion::execution::dataframe_impl
in datafusion-examples/examples/custom_datasource.rs:24:28

datafusion-examples/examples/custom_datasource.rs isn't present in
datafusion-examples.

Could the build process be using an old version of datafusion-examples module?

venkata.chaganti
revise dataframe.rs
47af10a
Change-Id: I3aea55073f454d9fd096df0d5d9bf7fe5533bfaa
@xudong963

xudong963 commented Mar 14, 2022•
edited
Loading

Copy link
Copy Markdown
Member

datafusion-examples/examples/custom_datasource.rs isn't present in
datafusion-examples.

It exists in master
image

However, the ticket didn't delete it
image

strange...

vchagand others added 4 commits March 14, 2022 14:03
@vchag
Merge branch 'apache:master' into merge_dataframe_and_dataframeImp
78dddc4
venkata.chaganti
fix import statements
fae8b34
Change-Id: I72f7bf086cf8a5726490b9cdf3bd52ff4154f006
venkata.chaganti
re-formatting files
8a818bc
Change-Id: Ib585edb0b6c7416c48a77168d82989e309dd776b
venkata.chaganti
bug fix
21621b4
Change-Id: I1af2eb7609912074b37e38f66c15d0ce75ed291a
@vchag

vchag commented Mar 14, 2022

Copy link
Copy Markdown
ContributorAuthor

datafusion-examples/examples/custom_datasource.rs isn't present in
datafusion-examples.

It exists in master image

However, the ticket didn't delete it image

strange...

Sorry, my bad. My branch was lagging. Merging apache: master into my branch fixed the issue.

xudong963
xudong963 approved these changes Mar 14, 2022
View reviewed changes

@xudong963xudong963 left a comment •
edited
Loading

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks again @vchag

cc @houqp

yjshen
yjshen approved these changes Mar 15, 2022
View reviewed changes

@yjshenyjshen left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @vchag! Very neat refactoring!

@yjshen
yjshen merged commit 7158b4b into apache:masterMar 15, 2022
@yjshen

yjshen commented Mar 15, 2022

Copy link
Copy Markdown
Member

Thanks @vchag@xudong963 !

@vchag

vchag commented Mar 15, 2022

Copy link
Copy Markdown
ContributorAuthor

Thanks @vchag@xudong963 !

yw!

@houqp

houqp commented Mar 17, 2022

Copy link
Copy Markdown
Member

nice work @vchag !

@houqphouqp mentioned this pull request Mar 17, 2022
Closed
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@yjshenyjshenyjshen approved these changes
@xudong963xudong963xudong963 approved these changes

Assignees

No one assigned

Labels

api changeChanges the API exposed to users of the crate

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Merge Dataframe trait and DataframeImpl into Dataframe struct

4 participants

@vchag@xudong963@yjshen@houqp