Skip to content

[fix](fe) Clear warmup jobs on available backends - #62931

Merged
gavinchou merged 1 commit into
apache:masterfrom
bobhan1:fix-warm-up-not-cancel
Apr 30, 2026
Merged

[fix](fe) Clear warmup jobs on available backends#62931
gavinchou merged 1 commit into
apache:masterfrom
bobhan1:fix-warm-up-not-cancel

Conversation

@bobhan1

Copy link
Copy Markdown
Contributor

What problem does this PR solve?

Issue Number: None

Related PR: None

Problem Summary: CORE-6050 reports that canceling a cloud warm up job can fail to notify surviving BEs when client initialization fails on an unavailable BE. Keep the normal warmup RPC path fail-fast, but make CLEAR_JOB client initialization skip unavailable BEs and continue clearing reachable BEs.

Release note

None

Check List (For Author)

  • Test: Unit Test
    • ./run-fe-ut.sh --run org.apache.doris.cloud.CloudWarmUpJobTest
  • Behavior changed: Yes. Cloud warm up cancellation now best-effort clears reachable BEs even if another BE is unavailable.
  • Does this need documentation: No

What problem does this PR solve?

Issue Number: close #xxx

Related PR: #xxx

Problem Summary:

Release note

None

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason
  • Behavior changed:

    • No.
    • Yes.
  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

Issue Number: None
Related PR: None
Problem Summary: CORE-6050 reports that canceling a cloud warm up job can fail to notify surviving BEs when client initialization fails on an unavailable BE. Keep the normal warmup RPC path fail-fast, but make CLEAR_JOB client initialization skip unavailable BEs and continue clearing reachable BEs.
None
- Test: Unit Test
- ./run-fe-ut.sh --run org.apache.doris.cloud.CloudWarmUpJobTest
- Behavior changed: Yes. Cloud warm up cancellation now best-effort clears reachable BEs even if another BE is unavailable.
- Does this need documentation: No
@bobhan1
bobhan1force-pushed the fix-warm-up-not-cancel branch from 53aa8b7 to 7374890CompareApril 29, 2026 07:13
@bobhan1

Copy link
Copy Markdown
ContributorAuthor

run buildall

@bobhan1

Copy link
Copy Markdown
ContributorAuthor

run nonConcurrent

@gavinchou

Copy link
Copy Markdown
Contributor

/review

@github-actionsgithub-actionsBot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review result: no blocking issues found.

Critical checkpoint conclusions:

  • Goal and proof: The PR keeps normal warm-up RPC client initialization fail-fast while making CLEAR_JOB best-effort across reachable BEs. The implementation matches that goal, and the added unit tests cover both the fail-fast SET_JOB path and the skip-unavailable CLEAR_JOB path.
  • Scope and clarity: The change is small and localized to CloudWarmUpJob client initialization plus targeted unit tests.
  • Concurrency and locking: cancel() and run() remain synchronized; the change does not add new shared state or new lock ordering. Existing RPCs still happen under the same synchronized job-level flow as before.
  • Lifecycle: Borrowed clients are returned on successful paths and invalidated/removed on failed clear RPC paths; partially initialized normal warm-up clients are released on failure.
  • Configuration/compatibility: No new configuration, storage format, thrift protocol, or FE-BE compatibility change.
  • Parallel paths: The PR intentionally preserves fail-fast behavior for SET_JOB/normal warm-up and changes only CLEAR_JOB cleanup behavior.
  • Error handling: Initialization failures during clear are logged per BE and do not block clearing other BEs; normal init still propagates failure.
  • Tests: Added FE unit coverage is relevant. I attempted ./run-fe-ut.sh --run org.apache.doris.cloud.CloudWarmUpJobTest, but this runner could not generate sources because thirdparty/installed/bin/protoc was missing.
  • Observability: New warning log includes BE id/address and job id for skipped clear initialization failures.
  • Transaction/persistence/data correctness: No transaction, persistence, version visibility, or data-write path changes.
  • Performance: No new hot-path loops or avoidable expensive work beyond existing per-BE client initialization/RPC behavior.

User focus: No additional user-provided review focus was present.

@github-actions

Copy link
Copy Markdown
Contributor

PR approved by at least one committer and no changes requested.

@github-actionsgithub-actionsBot added approved Indicates a PR has been approved by one committer. reviewed labels Apr 30, 2026
@github-actions

Copy link
Copy Markdown
Contributor

PR approved by anyone and no changes requested.

@gavinchou
gavinchou merged commit 5efc7f4 into apache:masterApr 30, 2026
34 checks passed
yiguolei pushed a commit that referenced this pull request May 7, 2026
#63029)
pick #62931
Problem Summary: CORE-6050 reports that canceling a cloud warm up job
can fail to notify surviving BEs when client initialization fails on an
unavailable BE. Keep the normal warmup RPC path fail-fast, but make
CLEAR_JOB client initialization skip unavailable BEs and continue
clearing reachable BEs.
### What problem does this PR solve?
Issue Number: close #xxx
Related PR: #xxx
Problem Summary:
### Release note
None
### Check List (For Author)
- Test <!-- At least one of them must be included. -->
- [ ] Regression test
- [ ] Unit Test
- [ ] Manual test (add detailed scripts or steps below)
- [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
- [ ] Previous test can cover this change.
- [ ] No code files have been changed.
- [ ] Other reason <!-- Add your reason? -->
- Behavior changed:
- [ ] No.
- [ ] Yes. <!-- Explain the behavior change -->
- Does this need documentation?
- [ ] No.
- [ ] Yes. <!-- Add document PR link here. eg:
apache/doris-website#1214 -->
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label <!-- Add branch pick label that this PR
should merge into -->
@yiguoleiyiguolei mentioned this pull request May 20, 2026
@morningmanmorningman mentioned this pull request Jun 3, 2026
zhaorongsheng pushed a commit to zhaorongsheng/doris that referenced this pull request Jun 4, 2026
Problem Summary: CORE-6050 reports that canceling a cloud warm up job
can fail to notify surviving BEs when client initialization fails on an
unavailable BE. Keep the normal warmup RPC path fail-fast, but make
CLEAR_JOB client initialization skip unavailable BEs and continue
clearing reachable BEs.
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

approvedIndicates a PR has been approved by one committer.dev/4.0.6-mergeddev/4.1.1-mergedint/3.1.6-mergedreviewed

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants

@bobhan1@gavinchou@morningman@yiguolei