Skip to content

Core: Extend ResolvingFileIO to support BulkOperations - #7976

Merged
nastra merged 10 commits into
apache:masterfrom
dramaticlly:resolveFileIO
Aug 14, 2023
Merged

Core: Extend ResolvingFileIO to support BulkOperations#7976
nastra merged 10 commits into
apache:masterfrom
dramaticlly:resolveFileIO

Conversation

@dramaticlly

@dramaticllydramaticlly commented Jul 3, 2023

Copy link
Copy Markdown
Contributor

close#7975

Since #6682 provide major deletion performance improvement on S3 and hadoop fileIO, I think we can try to add SupportsBulkOperations for ResolvingFileIO as it currently dynamically load fileIO based on scheme and both of the delegation fileIOs (S3FileIO and HadoopFileIO) supports bulk deletion.

If custom fileIO is used at catalog but not support bulk operations, it will fall back to standup sequential deletion.

CC @rdblue@nastra@RussellSpitzer@szehon-ho

nastra
nastra reviewed Aug 4, 2023
View reviewed changes
Comment threadcore/src/test/java/org/apache/iceberg/io/TestResolvingIO.java Outdated
nastra
nastra reviewed Aug 4, 2023
View reviewed changes

@nastranastra left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think having this makes a lot of sense, just left a small comment

bryanck
bryanck reviewed Aug 4, 2023
View reviewed changes
Comment threadcore/src/main/java/org/apache/iceberg/io/ResolvingFileIO.java Outdated
@bryanckbryanck mentioned this pull request Aug 4, 2023
Closed
dramaticllyand others added 2 commits August 4, 2023 11:07
RussellSpitzer
RussellSpitzer reviewed Aug 8, 2023
View reviewed changes
Comment threadcore/src/main/java/org/apache/iceberg/io/ResolvingFileIO.java Outdated
RussellSpitzer
RussellSpitzer reviewed Aug 8, 2023
View reviewed changes
Comment threadcore/src/test/java/org/apache/iceberg/io/TestResolvingIO.java
@bryanckbryanck mentioned this pull request Aug 9, 2023
Open
bryanck
bryanck reviewed Aug 10, 2023
View reviewed changes
Comment threadcore/src/main/java/org/apache/iceberg/io/ResolvingFileIO.java Outdated
nastra
nastra reviewed Aug 10, 2023
View reviewed changes
Comment threadcore/src/test/java/org/apache/iceberg/io/TestResolvingIO.java Outdated
nastra
nastra approved these changes Aug 10, 2023
View reviewed changes

@nastranastra left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM with one comment

RussellSpitzer
RussellSpitzer reviewed Aug 10, 2023
View reviewed changes
Comment threadcore/src/test/java/org/apache/iceberg/io/TestResolvingIO.java Outdated
RussellSpitzer
RussellSpitzer reviewed Aug 10, 2023
View reviewed changes
Comment threadcore/src/test/java/org/apache/iceberg/io/TestResolvingIO.java Outdated
@dramaticlly

dramaticlly commented Aug 10, 2023

Copy link
Copy Markdown
ContributorAuthor

@RussellSpitzer@nastra@bryanck updated per your suggest, appreciate another look.

bryanck
bryanck reviewed Aug 10, 2023
View reviewed changes
Comment threadcore/src/main/java/org/apache/iceberg/io/ResolvingFileIO.java Outdated
nastra
nastra reviewed Aug 11, 2023
View reviewed changes
Comment threadcore/src/test/java/org/apache/iceberg/io/TestResolvingIO.java Outdated
@nastra
nastra merged commit f4d830b into apache:masterAug 14, 2023
nastra pushed a commit to nastra/iceberg that referenced this pull request Aug 15, 2023
@bryanck@dramaticlly
Core: Extend ResolvingFileIO to support BulkOperations (apache#7976) (#…
69ca9cb
…139)
Co-authored-by: Hongyue/Steve Zhang <steveiszhy@gmail.com>
@dramaticlly
dramaticlly deleted the resolveFileIO branch January 26, 2025 04:06
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@nastranastranastra approved these changes
@RussellSpitzerRussellSpitzerRussellSpitzer left review comments
@bryanckbryanckbryanck approved these changes

Assignees

No one assigned

Labels

core

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

SupportsBulkOperations for ResolvingFileIO

4 participants

@dramaticlly@nastra@RussellSpitzer@bryanck