Skip to content

[Fix](recycler) Fix recycler Hdfs accessor list all bug - #46393

Merged
gavinchou merged 6 commits into
apache:masterfrom
Yukang-Lian:Fix-Recycler-Hdfs-List-All
Jan 13, 2025
Merged

[Fix](recycler) Fix recycler Hdfs accessor list all bug#46393
gavinchou merged 6 commits into
apache:masterfrom
Yukang-Lian:Fix-Recycler-Hdfs-List-All

Conversation

@Yukang-Lian

@Yukang-LianYukang-Lian commented Jan 3, 2025

Copy link
Copy Markdown
Collaborator

What problem does this PR solve?

Issue Number: close #xxx

Related PR: #xxx

Problem Summary:

The HDFS accessor in the recycler currently lists all files from the root directory when deleting by prefix, and then matches and deletes files based on the prefix. This approach wastes significant time processing unrelated tablets. This PR fixes the issue by only listing files under the relevant tablet directories first, then matching them against the prefix for deletion.

Release note

None

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason

  • Behavior changed:

    • No.
    • Yes.

  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@Thearas

Thearas commented Jan 3, 2025

Copy link
Copy Markdown
Contributor

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@dataroaringdataroaring added the usercase Important user case type label label Jan 3, 2025
dataroaring
dataroaring previously approved these changes Jan 3, 2025
View reviewed changes

@dataroaringdataroaring left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actionsgithub-actionsBot added the approved Indicates a PR has been approved by one committer. label Jan 3, 2025
@github-actions

github-actionsBot commented Jan 3, 2025

Copy link
Copy Markdown
Contributor

PR approved by at least one committer and no changes requested.

@github-actions

github-actionsBot commented Jan 3, 2025

Copy link
Copy Markdown
Contributor

PR approved by anyone and no changes requested.

@Yukang-Lian

Yukang-Lian commented Jan 3, 2025

Copy link
Copy Markdown
CollaboratorAuthor

run buildall

@github-actionsgithub-actionsBot removed the approved Indicates a PR has been approved by one committer. label Jan 3, 2025
gavinchou
gavinchou reviewed Jan 3, 2025
View reviewed changes
Comment threadcloud/src/recycler/hdfs_accessor.cpp Outdated
gavinchou
gavinchou reviewed Jan 3, 2025
View reviewed changes
Comment threadcloud/src/recycler/hdfs_accessor.cpp
gavinchou
gavinchou requested changes Jan 3, 2025
View reviewed changes
Comment threadcloud/src/recycler/hdfs_accessor.cpp Outdated
@Yukang-Lian
Yukang-Lianforce-pushed the Fix-Recycler-Hdfs-List-All branch from 41f1142 to 52edc39CompareJanuary 9, 2025 12:18
@Yukang-Lian

Yukang-Lian commented Jan 9, 2025

Copy link
Copy Markdown
CollaboratorAuthor

run buildall

@Yukang-Lian

Yukang-Lian commented Jan 9, 2025

Copy link
Copy Markdown
CollaboratorAuthor

run buildall

@Yukang-Lian

Yukang-Lian commented Jan 10, 2025

Copy link
Copy Markdown
CollaboratorAuthor

run buildall

gavinchou
gavinchou reviewed Jan 10, 2025
View reviewed changes
Comment threadcloud/src/recycler/hdfs_accessor.cpp
@Yukang-Lian

Yukang-Lian commented Jan 10, 2025

Copy link
Copy Markdown
CollaboratorAuthor

run buildall

@github-actions

github-actionsBot commented Jan 13, 2025

Copy link
Copy Markdown
Contributor

PR approved by at least one committer and no changes requested.

@github-actionsgithub-actionsBot added the approved Indicates a PR has been approved by one committer. label Jan 13, 2025
@gavinchou
gavinchou merged commit fff81ec into apache:masterJan 13, 2025
github-actionsBot pushed a commit that referenced this pull request Jan 13, 2025
@Yukang-LianYour Name
[Fix](recycler) Fix recycler Hdfs accessor list all bug (#46393)
d4bc09e
The HDFS accessor in the recycler currently lists all files from the
root directory when deleting by prefix, and then matches and deletes
files based on the prefix. This approach wastes significant time
processing unrelated tablets. This PR fixes the issue by only listing
files under the relevant tablet directories first, then matching them
against the prefix for deletion.
@github-actionsgithub-actionsBot mentioned this pull request Jan 13, 2025
Merged
dataroaring pushed a commit that referenced this pull request Jan 13, 2025
@github-actions@Yukang-Lian
branch-3.0: [Fix](recycler) Fix recycler Hdfs accessor list all bug#…
e1cb7ca
…46393 (#46918)
Cherry-picked from #46393
Co-authored-by: abmdocrt <lianyukang@selectdb.com>
@gavinchougavinchou mentioned this pull request Feb 18, 2025
Closed
lzyy2024 pushed a commit to lzyy2024/doris that referenced this pull request Feb 21, 2025
@Yukang-Lian@lzyy2024
[Fix](recycler) Fix recycler Hdfs accessor list all bug (apache#46393)
4e9e09b
The HDFS accessor in the recycler currently lists all files from the
root directory when deleting by prefix, and then matches and deletes
files based on the prefix. This approach wastes significant time
processing unrelated tablets. This PR fixes the issue by only listing
files under the relevant tablet directories first, then matching them
against the prefix for deletion.
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@gavinchougavinchougavinchou approved these changes
@luwei16luwei16luwei16 approved these changes
@dataroaringdataroaringdataroaring left review comments

Assignees

No one assigned

Labels

approvedIndicates a PR has been approved by one committer.dev/3.0.4-mergedp0_creviewedusercaseImportant user case type label

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@Yukang-Lian@Thearas@gavinchou@luwei16@dataroaring