Skip to content

branch-4.0: [opt](cloud) Add warm up job count metric on BE - #66137

Closed
liaoxin01 wants to merge 2 commits into
apache:branch-4.0from
liaoxin01:codex/pick-64734-to-branch-4.0
Closed

branch-4.0: [opt](cloud) Add warm up job count metric on BE#66137
liaoxin01 wants to merge 2 commits into
apache:branch-4.0from
liaoxin01:codex/pick-64734-to-branch-4.0

Conversation

@liaoxin01

@liaoxin01liaoxin01 commented Jul 27, 2026

Copy link
Copy Markdown
Contributor

Backport #64734 to branch-4.0.

What problem does this PR solve?

Issue Number: None

Related PR: #64734

Problem Summary: Add the per-BE file_cache_warm_up_job_num bvar so operators can see how many warm-up jobs are currently held in BE memory. The count changes only on real job state transitions, so retry/replay and duplicate clear requests do not skew it. The lazy cancellation cleanup path also decrements the metric and removes the matching event filter.

Release note

Add the file_cache_warm_up_job_num BE metric.

Check List (For Author)

  • Test: Manual test
    • clang-format 16 dry-run check passed for be/src/cloud/cloud_warm_up_manager.cpp
    • git diff --check passed
    • Local build was not run because this target-branch worktree does not provide hooks/setup_worktree.sh or thirdparty/installed; CI is pending
  • Behavior changed: Yes. A new per-BE warm-up job count metric is exposed.
  • Does this need documentation: No

@liaoxin01
[feature](cloud) Add warm up job count metric on BE
afb3b09
Add a `file_cache_warm_up_job_num` bvar that tracks the number of warm
up jobs currently held in a BE's memory. It is incremented when FE
dispatches a new job to the BE (SET_JOB / SET_BATCH / event-driven
SET_JOB) and decremented when the job is cleared (CLEAR_JOB /
event-driven CLEAR_JOB).
The increment/decrement are gated on actual state transitions
(`_cur_job_id` going 0 -> job_id, and `_tablet_replica_cache.erase()`
returning > 0), so retries, FE failover replays, and duplicate
CLEAR_JOB requests do not skew the count.
@liaoxin01
[fix](cloud) Decrement warm up job count on lazy cancellation path
ead6a9d
Address review: the event-driven `_tablet_replica_cache` can also be
erased in `get_replica_info()` when FE reports a warm up job as
CANCELLED (lazy cleanup used when an explicit CLEAR_JOB is missed).
That erase did not decrement `g_file_cache_warm_up_job_num`, so the
metric could stay at 1 after the job is no longer held.
Decrement the metric when this erase actually removes an entry, and
erase the matching `_event_driven_filters` entry to stay consistent
with the explicit CLEAR_JOB path in set_event().
@hello-stephen

hello-stephen commented Jul 27, 2026

Copy link
Copy Markdown
Contributor

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@liaoxin01
liaoxin01 marked this pull request as ready for review July 28, 2026 09:10
CopilotAI review requested due to automatic review settings July 28, 2026 09:10
@liaoxin01

liaoxin01 commented Jul 28, 2026

Copy link
Copy Markdown
ContributorAuthor

run buildall

CopilotAI reviewed Jul 28, 2026
View reviewed changes

CopilotAI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR backports the addition of a per-BE bvar metric (file_cache_warm_up_job_num) to track how many file-cache warm-up jobs are currently retained in BE memory, covering both RPC-driven (SET_JOB/SET_BATCH) and event-driven warm-up flows.

Changes:

  • Add new BE metric file_cache_warm_up_job_num to track active/held warm-up jobs.
  • Increment/decrement the metric on real state transitions for SET_JOB/SET_BATCH, CLEAR_JOB, event-driven set/clear, and lazy cancellation cleanup.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment threadbe/src/cloud/cloud_warm_up_manager.cpp
@hello-stephen

hello-stephen commented Jul 28, 2026

Copy link
Copy Markdown
Contributor

BE UT Coverage Report

Increment line coverage 30.77% (4/13) 🎉

Increment coverage report
Complete coverage report

CategoryCoverage
Function Coverage53.60% (19615/36596)
Line Coverage36.66% (183657/501000)
Region Coverage33.20% (142764/429960)
Branch Coverage34.07% (61720/181178)

@hello-stephen

hello-stephen commented Jul 28, 2026

Copy link
Copy Markdown
Contributor

BE Regression && UT Coverage Report

Increment line coverage 30.77% (4/13) 🎉

Increment coverage report
Complete coverage report

CategoryCoverage
Function Coverage71.51% (25603/35801)
Line Coverage54.48% (272049/499329)
Region Coverage52.09% (225753/433357)
Branch Coverage53.47% (97105/181590)

@hello-stephen

hello-stephen commented Jul 29, 2026

Copy link
Copy Markdown
Contributor

BE Regression && UT Coverage Report

Increment line coverage 30.77% (4/13) 🎉

Increment coverage report
Complete coverage report

CategoryCoverage
Function Coverage71.51% (25603/35801)
Line Coverage54.48% (272053/499329)
Region Coverage52.11% (225806/433357)
Branch Coverage53.47% (97103/181590)

@liaoxin01liaoxin01 reopened this Jul 30, 2026
@liaoxin01

liaoxin01 commented Jul 30, 2026

Copy link
Copy Markdown
ContributorAuthor

run buildall

@hello-stephen

hello-stephen commented Jul 30, 2026

Copy link
Copy Markdown
Contributor

BE Regression && UT Coverage Report

Increment line coverage 30.77% (4/13) 🎉

Increment coverage report
Complete coverage report

CategoryCoverage
Function Coverage71.35% (25567/35832)
Line Coverage54.26% (271026/499511)
Region Coverage51.86% (224715/433280)
Branch Coverage53.24% (96679/181595)

@hello-stephen

hello-stephen commented Jul 30, 2026

Copy link
Copy Markdown
Contributor

BE Regression && UT Coverage Report

Increment line coverage 30.77% (4/13) 🎉

Increment coverage report
Complete coverage report

CategoryCoverage
Function Coverage71.41% (25592/35836)
Line Coverage54.27% (271305/499936)
Region Coverage51.84% (224869/433760)
Branch Coverage53.23% (96759/181767)

@hello-stephen

hello-stephen commented Jul 30, 2026

Copy link
Copy Markdown
Contributor

BE Regression && UT Coverage Report

Increment line coverage 30.77% (4/13) 🎉

Increment coverage report
Complete coverage report

CategoryCoverage
Function Coverage71.53% (25634/35835)
Line Coverage54.45% (272236/499935)
Region Coverage52.04% (225740/433759)
Branch Coverage53.42% (97097/181767)

@hello-stephen

hello-stephen commented Jul 30, 2026

Copy link
Copy Markdown
Contributor

BE Regression && UT Coverage Report

Increment line coverage 30.77% (4/13) 🎉

Increment coverage report
Complete coverage report

CategoryCoverage
Function Coverage71.53% (25632/35835)
Line Coverage54.45% (272196/499935)
Region Coverage52.04% (225711/433759)
Branch Coverage53.41% (97080/181767)

@hello-stephen

hello-stephen commented Jul 30, 2026

Copy link
Copy Markdown
Contributor

BE Regression && UT Coverage Report

Increment line coverage 30.77% (4/13) 🎉

Increment coverage report
Complete coverage report

CategoryCoverage
Function Coverage71.53% (25632/35835)
Line Coverage54.45% (272207/499935)
Region Coverage52.04% (225733/433759)
Branch Coverage53.41% (97075/181767)

Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

Copilot code reviewCopilot
Copilot review effort, defaults to Lite
Applies to this pull request for everyone.Learn more about Copilot code review.
Copilot left review comments
@morningmanmorningmanAwaiting requested review from morningmanmorningman is a code owner

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@liaoxin01@hello-stephen