refactor: Move various parts of datasource out of core by logan-keede · Pull Request #14616 · apache/datafusion

logan-keede · 2025-02-11T20:34:12Z

Which issue does this PR close?

part of [Epic] Split datasources out from datafusion crate (datafusion/core) #14444.

Rationale for this change

What changes are included in this PR?

Move the parts of datasource that do not have much coupling out of core.

Are these changes tested?

yes, by Github CI.

Are there any user-facing changes?

No, there should not be.

…_filesink

logan-keede · 2025-02-11T20:42:44Z

cc @alamb

alamb

Thank you @logan-keede -- I think this looks great to me. I have a few small questions but nothing major that can't be done as a follow on PR

I love this incremental approach @logan-keede -- very very nice

alamb · 2025-02-12T01:03:36Z

+/// as other operators.
+///
+/// [`FileStream`]: <https://github.com/apache/datafusion/blob/main/datafusion/core/src/datasource/physical_plan/file_stream.rs>
+pub struct FileStreamMetrics {


I was thinking about if there was some better name for this module than file_stream_part.rs but I couldn't come up with one

Eventually, when all of file_stream is moved, we can change the name. or perhaps we should just keep it file_stream as I did not make a file_scan_config_part but file_scan_config (it was an arbitrary decision).

I prefer file_stream

alamb · 2025-02-12T01:04:35Z

 use crate::datasource::listing::{FileRange, ListingTableUrl, PartitionedFile};
-use crate::datasource::physical_plan::file_stream::{FileOpenFuture, FileOpener};
 use crate::datasource::physical_plan::FileMeta;
+use crate::datasource::physical_plan::{FileOpenFuture, FileOpener};


do you think it is ok to re-export FileOpenFuture and FileOpener in physical_plan ? Or should they still be in a file_stream submodule?

They are already in physical_plan. They could be imported from both file_stream and physical_plan earlier.
datafusion/datafusion/core/src/datasource/physical_plan/mod.rs
Line 58 in abb7669
pubuse file_stream::{FileOpenFuture,FileOpener,FileStream,OnError};
or did I misunderstand something?

jayzhan211 · 2025-02-12T06:08:40Z

Why not move to the datasource crate but catalog-listing

logan-keede · 2025-02-12T07:34:11Z

Why not move to the datasource crate but catalog-listing

original plan was to move just datasource::listing to catalog-listing, but listing/table.rs has some dependencies on 'these' files (almost all file with pattern "physical_plan/file_*.rs" or "file_format/file_*.rs") but 'these' files also have some dependency on catalog-listing (like PartitionedFile and more). So, this was all I could think of to avoid circular dependency and keeping original structure somewhat intact. do you still think we can make a datasource crate distinct from catalog-listing?

jayzhan211 · 2025-02-12T09:24:07Z

FileCompressionType, PartitionedFile, FileRange can be move to datasource.

If A and B is tightly couple, you need to pull partial structure out to C and import C for A and B. Not moving A and B together.

alamb · 2025-02-12T11:21:41Z

FileCompressionType, PartitionedFile, FileRange can be move to datasource.
If A and B is tightly couple, you need to pull partial structure out to C and import C for A and B. Not moving A and B together.

The possible plan I proposed on #14444 proposes a structure like this

datafusion-catalog-listing: ListingTable and associated types like PartitionedFile
datafusion-datasource-parquet: ParquetExec and file firmat
datafusion-datasource-avroAvroExec and file formats
datafusion-datasource-arrow
datafusion-datasource-json
datafusion-datasource-csv

This was before the refactor with DataSource landed:

Introduce unified DataSourceExec for provided datasources, remove ParquetExec, CsvExec, etc #14224

@jayzhan211 are you proposing we add another crate in there, something like?

datafusion-catalog-listing: ListingTable and associated types like PartitionedFile
datafusion-datasource: NEW that holds FileCompressionType, PartitonedFile,and DataSource
datafusion-datasource-parquet: ParquetExec and file firmat
datafusion-datasource-avroAvroExec and file formats
datafusion-datasource-arrow
datafusion-datasource-json
datafusion-datasource-csv

alamb · 2025-02-12T11:22:05Z

FWIW I think we can merge this PR and then keep moving code around as follow on PRs)

jayzhan211 · 2025-02-12T11:50:38Z

Yes and not only the 3 I mentioned, FileFormat, FileFormatFactory, etc. I think File related structure should belong into it own crate and then the implementation like parquet, arrow or listing table import it.

jayzhan211 · 2025-02-12T11:55:15Z

Catalog -> Schema -> Table -> FileFormat -> QueryPlanner is the overall dependency w
e follow by.

jayzhan211 · 2025-02-12T12:13:24Z

 use arrow::error::ArrowError;
 use arrow::record_batch::RecordBatch;
-use datafusion_common::instant::Instant;
+pub use datafusion_catalog_listing::file_stream_part::*;


It's better to avoid importing everything at once so that the imported modules are explicitly controlled and managed.

logan-keede · 2025-02-12T12:59:31Z

mostly for my clarity

also, I am not suggesting that we keep the file_format and physical_plan folders in new solution,
though I was going to suggest(only one folder for all file related) if we move forward by old approach

PS: @jayzhan211 can/should I tag you in related PRs?

alamb · 2025-02-12T14:35:53Z

also, I am not suggesting that we keep the file_format and physical_plan folders in new solution,
though I was going to suggest(only one folder for all file related) if we move forward by old approach

That is my understanding as well

alamb · 2025-02-12T15:00:45Z

Ok, I am going to merge this one in and we can keep working on this in follow on PRs.

@logan-keede can you update the plan on #14444 and keep on hacking ?

Thanks again!

logan-keede added 9 commits February 11, 2025 18:43

exploration with cargo machete
4033d56

readdition
15d2a4d

more dependency removals
bc10844

fix: ci
5786e96

fix: format
e8d50e4

revert unnecessary
b546794

First Iteration
9dd90dc

Merge branch 'main' ofhttps://github.com/apache/datafusioninto move…
d85359d
…_filesink

fix: forgotten Header
cc61f6a

github-actionsBot added the core Core DataFusion crate label Feb 11, 2025

alamb approved these changes Feb 12, 2025
View reviewed changes

jayzhan211 approved these changes Feb 12, 2025
View reviewed changes

jayzhan211 reviewed Feb 12, 2025
View reviewed changes

Tweaks
709e0f6

alamb merged commit 980931c into apache:mainFeb 12, 2025

alamb mentioned this pull request Feb 12, 2025
[Epic] Split datasources out from datafusion crate (datafusion/core) #14444
Closed

logan-keede mentioned this pull request Feb 13, 2025
refactor: Make catalog datasource #14643
Merged

Uh oh!

Conversation

logan-keede commented Feb 11, 2025• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Which issue does this PR close?

Rationale for this change

What changes are included in this PR?

Are these changes tested?

Are there any user-facing changes?

Uh oh!

logan-keede commented Feb 11, 2025

Uh oh!

alamb left a comment

Choose a reason for hiding this comment

Uh oh!

alambFeb 12, 2025

Choose a reason for hiding this comment

Uh oh!

logan-keedeFeb 12, 2025

Choose a reason for hiding this comment

Uh oh!

jayzhan211Feb 12, 2025

Choose a reason for hiding this comment

Uh oh!

alambFeb 12, 2025

Choose a reason for hiding this comment

Uh oh!

logan-keedeFeb 12, 2025

Choose a reason for hiding this comment

Uh oh!

jayzhan211 commented Feb 12, 2025

Uh oh!

logan-keede commented Feb 12, 2025

Uh oh!

jayzhan211 commented Feb 12, 2025

Uh oh!

alamb commented Feb 12, 2025

Uh oh!

alamb commented Feb 12, 2025

Uh oh!

jayzhan211 commented Feb 12, 2025

Uh oh!

jayzhan211 commented Feb 12, 2025• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jayzhan211Feb 12, 2025

Choose a reason for hiding this comment

Uh oh!

logan-keede commented Feb 12, 2025

Uh oh!

alamb commented Feb 12, 2025

Uh oh!

alamb commented Feb 12, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

logan-keede commented Feb 11, 2025•
edited
Loading

jayzhan211 commented Feb 12, 2025•
edited
Loading