Skip to content

[feature](Recycler) Parallelize s3 delete operations and recycle_tablet - #37630

Merged
gavinchou merged 1 commit into
apache:masterfrom
ByteYue:batch_delete_for_recycler
Jul 19, 2024
Merged

[feature](Recycler) Parallelize s3 delete operations and recycle_tablet #37630
gavinchou merged 1 commit into
apache:masterfrom
ByteYue:batch_delete_for_recycler

Conversation

@ByteYue

@ByteYueByteYue commented Jul 10, 2024•
edited
Loading

Copy link
Copy Markdown
Contributor

Proposed changes

Previously the procedure of recycler instance is single-threaded, which is not full sufficiently parallel. And there exists many network IO operation. So this pr tries to spilt recycle tasks into different stage which can be parallel. And make the delete operations parallel.

@doris-robot

doris-robot commented Jul 10, 2024

Copy link
Copy Markdown

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@ByteYue
ByteYueforce-pushed the batch_delete_for_recycler branch from cdeabe3 to 0d1bc2aCompareJuly 10, 2024 15:24
github-actions[bot]
github-actionsBot reviewed Jul 11, 2024
View reviewed changes

@github-actionsgithub-actionsBot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

clang-tidy made some suggestions

Comment threadcloud/src/recycler/obj_storage_client.cpp
Comment threadcloud/src/recycler/util.h Outdated
github-actions[bot]
github-actionsBot reviewed Jul 12, 2024
View reviewed changes

@github-actionsgithub-actionsBot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

clang-tidy made some suggestions

Comment threadcloud/test/recycler_test.cpp
Comment threadcloud/test/recycler_test.cpp Outdated
@ByteYue
ByteYueforce-pushed the batch_delete_for_recycler branch from 217d892 to a323207CompareJuly 12, 2024 07:23
@ByteYue

ByteYue commented Jul 12, 2024

Copy link
Copy Markdown
ContributorAuthor

run buildall

@ByteYueByteYue changed the title [feature-wip](Recycler) Parallelize s3 delete operations and recycle_tablet [feature](Recycler) Parallelize s3 delete operations and recycle_tablet Jul 12, 2024
github-actions[bot]
github-actionsBot reviewed Jul 12, 2024
View reviewed changes

@github-actionsgithub-actionsBot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

clang-tidy made some suggestions

Comment threadcloud/test/util_test.cpp
github-actions[bot]
github-actionsBot reviewed Jul 12, 2024
View reviewed changes

@github-actionsgithub-actionsBot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

clang-tidy made some suggestions

Comment threadcloud/test/util_test.cpp
@ByteYue

ByteYue commented Jul 12, 2024

Copy link
Copy Markdown
ContributorAuthor

run buildall

@ByteYue
ByteYueforce-pushed the batch_delete_for_recycler branch from 13f3b11 to 7dd23dcCompareJuly 15, 2024 10:48
@ByteYue

ByteYue commented Jul 15, 2024

Copy link
Copy Markdown
ContributorAuthor

run buildall

platoneko
platoneko requested changes Jul 16, 2024
View reviewed changes
Comment threadcloud/src/recycler/s3_accessor.h Outdated
Comment threadcloud/src/recycler/s3_accessor.cpp Outdated
Comment threadcloud/src/recycler/recycler.h Outdated
Comment threadcloud/src/recycler/util.h Outdated
Comment threadcloud/src/recycler/util.h
Comment threadcloud/src/recycler/obj_storage_client.h Outdated
github-actions[bot]
github-actionsBot reviewed Jul 16, 2024
View reviewed changes

@github-actionsgithub-actionsBot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

clang-tidy made some suggestions

Comment threadcloud/src/recycler/sync_executor.h
Comment threadcloud/src/recycler/util.h
@ByteYue

ByteYue commented Jul 16, 2024

Copy link
Copy Markdown
ContributorAuthor

run buildall

@ByteYue
ByteYueforce-pushed the batch_delete_for_recycler branch from 19c6e5b to 4baa42fCompareJuly 16, 2024 10:22
@ByteYue

ByteYue commented Jul 16, 2024

Copy link
Copy Markdown
ContributorAuthor

run buildall

@ByteYue
ByteYueforce-pushed the batch_delete_for_recycler branch 4 times, most recently from 09d8af2 to f605509CompareJuly 17, 2024 08:24
@ByteYue

ByteYue commented Jul 17, 2024

Copy link
Copy Markdown
ContributorAuthor

run buildall

@ByteYue
ByteYueforce-pushed the batch_delete_for_recycler branch from f605509 to 2c12345CompareJuly 18, 2024 03:00
@ByteYue
ByteYue requested a review from platonekoJuly 18, 2024 03:00
@ByteYue

ByteYue commented Jul 18, 2024

Copy link
Copy Markdown
ContributorAuthor

run buildall

@ByteYue
add util test
6f5219d
use atomic
use value
@ByteYue
ByteYueforce-pushed the batch_delete_for_recycler branch from 2c12345 to 6f5219dCompareJuly 19, 2024 07:50
@ByteYue

ByteYue commented Jul 19, 2024

Copy link
Copy Markdown
ContributorAuthor

run buildall

@github-actions

github-actionsBot commented Jul 19, 2024

Copy link
Copy Markdown
Contributor

PR approved by at least one committer and no changes requested.

@github-actionsgithub-actionsBot added approved Indicates a PR has been approved by one committer. reviewed labels Jul 19, 2024
@github-actions

github-actionsBot commented Jul 19, 2024

Copy link
Copy Markdown
Contributor

PR approved by anyone and no changes requested.

@gavinchou
gavinchou merged commit 05b05bd into apache:masterJul 19, 2024
dataroaring pushed a commit that referenced this pull request Aug 16, 2024
@ByteYue@dataroaring
[feature](Recycler) Parallelize s3 delete operations and recycle_tabl…
0705bfa
…et (#37630)
Previously the procedure of recycler instance is single-threaded, which
is not full sufficiently parallel. And there exists many network IO
operation. So this pr tries to spilt recycle tasks into different stage
which can be parallel. And make the delete operations parallel.
@gavinchougavinchou mentioned this pull request Oct 13, 2024
Closed
HappenLee pushed a commit to HappenLee/incubator-doris that referenced this pull request Apr 24, 2026
@ByteYue@HappenLee
[feature](Recycler) Parallelize s3 delete operations and recycle_tabl…
7af8cc6
…et (apache#37630)
Previously the procedure of recycler instance is single-threaded, which
is not full sufficiently parallel. And there exists many network IO
operation. So this pr tries to spilt recycle tasks into different stage
which can be parallel. And make the delete operations parallel.
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@github-actionsgithub-actions[bot]github-actions[bot] left review comments
@gavinchougavinchougavinchou approved these changes
@platonekoplatonekoplatoneko approved these changes

Assignees

No one assigned

Labels

approvedIndicates a PR has been approved by one committer.dev/3.0.2-mergedreviewed

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@ByteYue@doris-robot@gavinchou@platoneko@dataroaring