Uh oh!
There was an error while loading. Please reload this page.
[fix](fe) Cancel rebuilt VCG warm up jobs on drop - #65426
Conversation
hello-stephen
commented
Jul 9, 2026
Thank you for your contribution to Apache Doris. Please clearly describe your PR:
|
deardeng
commented
Jul 13, 2026
run buildall |
hello-stephen
commented
Jul 13, 2026
TPC-H: Total hot run time: 29784 ms |
hello-stephen
commented
Jul 13, 2026
TPC-DS: Total hot run time: 179720 ms |
hello-stephen
commented
Jul 13, 2026
ClickBench: Total hot run time: 25.2 s |
hello-stephen
commented
Jul 13, 2026
FE UT Coverage ReportIncrement line coverage |
### What problem does this PR solve? Issue Number: closeapache#26980 Related PR: apache#65153 Problem Summary: When active/standby switches for a virtual compute group, FE rebuilds periodic and event-driven file cache warm up jobs and writes new job ids to MS. FE memory kept stale cacheWarmupJobIds until a later MS sync, so a quick DROP VCG cancelled old ids and left rebuilt jobs running. This change makes updateFileCacheJobIds report MS update success, synchronizes the FE-local policy with rebuilt job ids after a successful update, and cancels newly created jobs if the MS update fails. ### Release note Fixes VCG drop cleanup after warm up job rebuild. ### Check List (For Author) - Test: Unit Test - ./run-fe-ut.sh --run org.apache.doris.cloud.catalog.CloudInstanceStatusCheckerTest - Behavior changed: Yes. Drop VCG now cancels rebuilt warm up jobs after active/standby switch. - Does this need documentation: No
75f8682 to
6ff038bComparedeardeng
commented
Jul 23, 2026
run buildall |
hello-stephen
commented
Jul 23, 2026
TPC-H: Total hot run time: 29097 ms |
hello-stephen
commented
Jul 23, 2026
TPC-DS: Total hot run time: 177439 ms |
PR approved by at least one committer and no changes requested. |
PR approved by anyone and no changes requested. |
hello-stephen
commented
Jul 23, 2026
ClickBench: Total hot run time: 24.99 s |
deardeng
commented
Jul 24, 2026
/review |
There was a problem hiding this comment.
Automated review result: request changes.
The happy path now keeps FE-local job IDs aligned after an OK MetaService response, and the new tests cover fast VCG drop plus explicit publication failure. Three failure/concurrency issues remain.
Critical checkpoint conclusions:
- Goal and proof: the normal switch/drop case is covered, but crash/replay, concurrent switch, and sustained publication-failure behavior are not safe; see the inline findings.
- Scope and clarity: the diff is focused, but job creation, MS publication, FE-local state, and retry are still handled as separate state transitions without a durable pending-publication abstraction.
- Concurrency: routing threads can switch active/standby concurrently with the checker daemon; no policy epoch or object-field lock protects the IDs/rebuild flag (P2 finding).
- Lifecycle and persistence: warm-up jobs are edit-logged before their IDs are published, and no startup reconciliation handles live jobs absent from the MS policy (P1 finding).
- Configuration: no new configuration was added. Existing 10-second checker and seven-day history defaults amplify the failure retry path.
- Compatibility: no storage, wire-format, or FE-BE protocol change was introduced; the Java return-type change has one in-artifact production caller and it is updated.
- Parallel/conditional paths: OK, non-OK, RpcException, VCG drop, failover, replay, and retry paths were traced. The boolean classification itself is correct, but successful rollback retries recreate durable jobs every cycle (P1 finding).
- Tests/results: added JUnit tests cover success/drop and explicit false rollback. Latch-based switch concurrency, master-exit/replay, and sustained-failure retry tests are missing. Live COMPILE, FE UT, CheckStyle, and cloud_p0 checks pass. check_coverage_fe failed before fe-core ran because Maven received a truncated icu4j download; it is an infrastructure failure, not evidence of a coverage regression. No local builds/tests were run under the review-only runner contract.
- Observability: the new logs include VCG, cluster, and job identifiers. However, failed retries create job-ID metrics with no removal path (covered by the retry-churn finding).
- Transactions/data writes: no user-data transaction path changes. FE edit-log job creation and MetaService policy publication are not atomic, which is the core lifecycle risk.
- FE-BE variables: none added or changed.
- Performance: the success path adds one local setter only; the sustained failure path causes large job-history, journal, and metric growth (P1 finding).
- Other applicable concerns: no additional configuration, security, storage-version, or documentation issue was found.
User focus: no additional user-provided focus points were supplied.
| // send jobIds to ms | ||
| List<String> newJobIds = Arrays.asList(Long.toString(jobIdPeriodic), Long.toString(jobIdEvent)); | ||
| CloudSystemInfoService.updateFileCacheJobIds(virtualGroupInFe, newJobIds); | ||
| boolean updated = CloudSystemInfoService.updateFileCacheJobIds(virtualGroupInFe, newJobIds); |
There was a problem hiding this comment.
[P1] Reconcile jobs when publication rollback cannot run
Both jobs are registered in CacheHotspotManager and edit-logged before this RPC. If the master exits/crashes after either creation but before publication and rollback finish, the next master replays live jobs that are absent from the MS policy; the next rebuild then hits repeatJobDetectionSet on the first same-key createJob and returns before it can ever publish replacement IDs. A failure of the second createJob similarly leaves the first job behind. Please track and clean up every successfully created ID on all later failures, and add new-master reconciliation for live VCG jobs that are not referenced by MS so this cannot wedge warmup indefinitely.
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
Problem Summary: When active/standby switches for a virtual compute group, FE rebuilds periodic and event-driven file cache warm up jobs and writes new job ids to MS. FE memory kept stale cacheWarmupJobIds until a later MS sync, so a quick DROP VCG cancelled old ids and left rebuilt jobs running. This change makes updateFileCacheJobIds report MS update success, synchronizes the FE-local policy with rebuilt job ids after a successful update, and cancels newly created jobs if the MS update fails.
pick from #65426 Problem Summary: When active/standby switches for a virtual compute group, FE rebuilds periodic and event-driven file cache warm up jobs and writes new job ids to MS. FE memory kept stale cacheWarmupJobIds until a later MS sync, so a quick DROP VCG cancelled old ids and left rebuilt jobs running. This change makes updateFileCacheJobIds report MS update success, synchronizes the FE-local policy with rebuilt job ids after a successful update, and cancels newly created jobs if the MS update fails. (cherry picked from commit 60e56ec) ### What problem does this PR solve? Issue Number: close #xxx Related PR: #xxx Problem Summary: ### Release note None ### Check List (For Author) - Test <!-- At least one of them must be included. --> - [ ] Regression test - [ ] Unit Test - [ ] Manual test (add detailed scripts or steps below) - [ ] No need to test or manual test. Explain why: - [ ] This is a refactor/code format and no logic has been changed. - [ ] Previous test can cover this change. - [ ] No code files have been changed. - [ ] Other reason <!-- Add your reason? --> - Behavior changed: - [ ] No. - [ ] Yes. <!-- Explain the behavior change --> - Does this need documentation? - [ ] No. - [ ] Yes. <!-- Add document PR link here. eg: apache/doris-website#1214 --> ### Check List (For Reviewer who merge this PR) - [ ] Confirm the release note - [ ] Confirm test cases - [ ] Confirm document - [ ] Add branch pick label <!-- Add branch pick label that this PR should merge into -->
Problem Summary: When active/standby switches for a virtual compute group, FE rebuilds periodic and event-driven file cache warm up jobs and writes new job ids to MS. FE memory kept stale cacheWarmupJobIds until a later MS sync, so a quick DROP VCG cancelled old ids and left rebuilt jobs running. This change makes updateFileCacheJobIds report MS update success, synchronizes the FE-local policy with rebuilt job ids after a successful update, and cancels newly created jobs if the MS update fails.
Problem Summary: When active/standby switches for a virtual compute group, FE rebuilds periodic and event-driven file cache warm up jobs and writes new job ids to MS. FE memory kept stale cacheWarmupJobIds until a later MS sync, so a quick DROP VCG cancelled old ids and left rebuilt jobs running. This change makes updateFileCacheJobIds report MS update success, synchronizes the FE-local policy with rebuilt job ids after a successful update, and cancels newly created jobs if the MS update fails.
What problem does this PR solve?
Issue Number: close #xxx
Related PR: #xxx
Problem Summary:
Release note
None
Check List (For Author)
Test
Behavior changed:
Does this need documentation?
Check List (For Reviewer who merge this PR)