[fix](job) Recover broker load from visible transaction by sollhui · Pull Request #66987 · apache/doris

sollhui · 2026-08-20T08:36:56Z

What problem does this PR solve?

In cloud mode, a Broker Load transaction can become VISIBLE in Meta Service before FE persists the final Load Job record. If FE restarts in that window, the replayed PENDING job begins the same label again and is cancelled after conflicting with its own visible transaction.

This change looks up the transaction by label, verifies that its callback ID belongs to the current Load Job, reuses PREPARE transactions, and finishes the job immediately when the transaction is already VISIBLE. For Cloud Broker Load, the final operation is persisted to EditLog.

Release note

Fix Broker Load jobs being reported as cancelled after FE restart even though their transactions were already visible.

Check List (For Author)

Test
- Regression test
- Unit Test
  - ./run-fe-ut.sh --run org.apache.doris.load.loadv2.BrokerLoadJobTest
- Manual test (add detailed scripts or steps below)
- No need to test or manual test. Explain why:
  - This is a refactor/code format and no logic has been changed.
  - Previous test can cover this change.
  - No code files have been changed.
  - Other reason
Behavior changed:
- No.
- Yes.
  - A replayed Broker Load adopts its own visible transaction and finishes successfully.
Does this need documentation?
- No.
- Yes.

Check List (For Reviewer who merge this PR)

Confirm the release note
Confirm test cases
Confirm document
Add branch pick label

…problem does this PR solve?\n\nIssue Number: N/A\n\nRelated PR: apache#66469\n\nProblem Summary: In cloud mode, a broker load transaction can become visible in meta service before FE persists the final load-job record. If FE restarts in that window, the replayed PENDING job begins the same label again and is cancelled after conflicting with its own visible transaction. Look up the label transaction, verify its callback belongs to the job, reuse PREPARE transactions, and finish the job from VISIBLE transactions while persisting the cloud final operation.\n\n### Release note\n\nFix broker load jobs being reported as cancelled after FE restart even though their transactions were already visible.\n\n### Check List (For Author)\n\n- Test: Unit Test\n - ./run-fe-ut.sh --run org.apache.doris.load.loadv2.BrokerLoadJobTest\n- Behavior changed: Yes. A replayed broker load adopts its own visible transaction and finishes successfully.\n- Does this need documentation: No

hello-stephen · 2026-08-20T08:37:01Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

What problem was fixed (it's best to include specific error reporting information). How it was fixed.
Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
What features were added. Why was this function added?
Which code was refactored and why was this part of the code refactored?
Which functions were optimized and what is the difference before and after the optimization?

sollhui · 2026-08-21T02:45:44Z

run buildall

sollhui · 2026-08-21T02:45:50Z

/review

github-actions

Review outcome: request changes. The branch-local PREPARE/VISIBLE/foreign handling is covered, but the real replay lifecycle still has five correctness and persistence failures described inline.

Checkpoint conclusions:

Goal and test proof: partially achieved; direct VISIBLE recovery works in the mock, but actual task ordering, durable COMMITTED, cancellation, and second replay remain unsafe.
Scope and quality: the production change is focused and reuses existing transaction APIs; no unrelated change was found.
Concurrency: attachment replay and FINISHED selection are not atomic with user cancellation.
Lifecycle and cleanup: a reached base-job recovery removes the callback and the later pending callback is ignored; the suspected CopyJob hook issue was dismissed because no current replayed CopyJob can reach this branch.
Configuration: no new option is added, but the existing default cloud lazy-commit setting exposes the COMMITTED failure.
Compatibility and parallel paths: no API, protocol, storage-format, FE-BE variable, or rolling-version surface changes; local/cloud status mapping and all production subclasses were checked.
Conditions and exceptions: null, foreign, aborted, and unknown transactions fail closed; source-discovery failures and owned COMMITTED are mishandled as noted inline.
Tests and results: the Mockito tests cover only direct helper branches and omit real source ordering, a persistent COMMITTED txn, a real attachment, cancellation, durable final fields, and a second replay. Per the review-only runner instruction, no build or test was run.
Observability and persistence: the adopted transaction ID is not replayable and recovered finish time is replaced with FE recovery time.
Transaction/data correctness: an already successful or irrevocably committed load can be durably represented as CANCELLED.
Performance: the added lookup is confined to the exceptional label-conflict path; no hot-path concern was found.
Other issues: three review rounds converged with no additional distinct finding.

No additional user-provided focus was supplied.

hello-stephen · 2026-08-21T04:17:12Z

FE UT Coverage Report

Increment line coverage 92.31% (12/13) 🎉
Increment coverage report
Complete coverage report

hello-stephen · 2026-08-21T04:36:57Z

TPC-H: Total hot run time: 17519 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit c3967b9b3b1640ae2a2107575fa7bcbc26c068b0, data reload: false
------ Round 1 ----------------------------------
============================================
q1	17551	3001	3010	3001
q2	1923	237	158	158
q3	10435	897	524	524
q4	4667	250	203	203
q5	7677	571	389	389
q6	139	118	93	93
q7	525	506	378	378
q8	9250	906	968	906
q9	3513	2427	2414	2414
q10	6530	890	706	706
q11	440	268	252	252
q12	695	384	338	338
q13	17865	1540	1186	1186
q14	161	158	145	145
q15	q16	446	407	368	368
q17	801	824	841	824
q18	3167	2335	2281	2281
q19	1162	940	833	833
q20	667	526	499	499
q21	5464	1788	1961	1788
q22	335	261	233	233
Total cold run time: 93413 ms
Total hot run time: 17519 ms
----- Round 2, with runtime_filter_mode=off -----
============================================
q1	3393	3318	3297	3297
q2	216	207	150	150
q3	2302	2356	2209	2209
q4	1185	1185	899	899
q5	2223	2161	2138	2138
q6	181	130	88	88
q7	1020	909	866	866
q8	1610	1408	1404	1404
q9	3158	3145	3142	3142
q10	1903	1834	1654	1654
q11	355	267	251	251
q12	456	430	337	337
q13	1488	1572	1162	1162
q14	181	185	161	161
q15	q16	395	396	356	356
q17	1036	1019	1026	1019
q18	5003	4442	4803	4442
q19	840	860	879	860
q20	978	950	807	807
q21	3755	3117	3204	3117
q22	395	354	319	319
Total cold run time: 32073 ms
Total hot run time: 28678 ms

hello-stephen · 2026-08-21T04:43:32Z

TPC-DS: Total hot run time: 83960 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit c3967b9b3b1640ae2a2107575fa7bcbc26c068b0, data reload: false
query5	4307	409	349	349
query6	410	163	160	160
query7	4856	471	265	265
query8	294	121	112	112
query9	8709	2919	2893	2893
query10	392	249	213	213
query11	5375	1056	946	946
query12	119	72	73	72
query13	1203	445	332	332
query14	6145	2227	2102	2102
query14_1	2005	1978	1989	1978
query15	177	127	115	115
query16	929	405	375	375
query17	809	456	373	373
query18	2344	333	245	245
query19	173	154	122	122
query20	75	72	70	70
query21	220	119	103	103
query22	5470	5370	5378	5370
query23	6992	6209	6102	6102
query23_1	6249	6332	6443	6332
query24	7316	1103	786	786
query24_1	797	813	784	784
query25	434	305	262	262
query26	1264	263	173	173
query27	2750	445	293	293
query28	4583	1504	1506	1504
query29	943	461	369	369
query30	289	181	156	156
query31	873	426	357	357
query32	102	51	52	51
query33	468	252	172	172
query34	973	856	504	504
query35	414	405	349	349
query36	561	559	506	506
query37	119	78	71	71
query38	1008	848	815	815
query39	497	477	458	458
query39_1	448	481	483	481
query40	222	121	109	109
query41	52	50	51	50
query42	79	80	80	80
query43	243	240	212	212
query44	1022	548	546	546
query45	107	109	103	103
query46	766	836	534	534
query47	752	763	717	717
query48	327	295	218	218
query49	532	236	186	186
query50	810	324	269	269
query51	8051	7986	7931	7931
query52	95	91	66	66
query53	206	213	158	158
query54	220	186	184	184
query55	73	66	59	59
query56	276	234	214	214
query57	699	670	622	622
query58	220	200	197	197
query59	1200	1227	1097	1097
query60	246	213	193	193
query61	141	108	141	108
query62	359	204	192	192
query63	185	167	158	158
query64	2763	703	551	551
query65	1671	1650	1649	1649
query66	1950	316	273	273
query67	10151	9854	9601	9601
query68	2751	1262	780	780
query69	347	217	199	199
query70	640	615	599	599
query71	288	268	236	236
query72	2294	1685	1555	1555
query73	665	613	362	362
query74	1573	1249	1149	1149
query75	1236	1171	1012	1012
query76	2290	734	546	546
query77	265	262	218	218
query78	3896	3566	3224	3224
query79	3009	858	597	597
query80	1599	430	397	397
query81	512	213	180	180
query82	658	176	101	101
query83	319	249	237	237
query84	314	119	105	105
query85	868	459	403	403
query86	468	169	160	160
query87	996	958	908	908
query88	2983	2145	2140	2140
query89	303	226	200	200
query90	2027	146	152	146
query91	154	144	133	133
query92	61	50	43	43
query93	1745	1259	756	756
query94	635	281	225	225
query95	612	377	346	346
query96	826	606	282	282
query97	1135	1090	1027	1027
query98	181	139	147	139
query99	419	352	306	306
Total cold run time: 179993 ms
Total hot run time: 83960 ms

### What problem does this PR solve? Issue Number: N/A Related PR: apache#66987 Problem Summary: The visible-transaction recovery added for broker load could race with cancellation and persisted neither the recovered transaction ID nor the Meta Service finish time. Make attachment replay and terminal-state selection atomic under the load-job lock, persist the recovered transaction ID in cloud end-load-job journals, and retain the transaction durable finish time. ### Release note None ### Check List (For Author) - Test: Unit Test - ./run-fe-ut.sh --run org.apache.doris.load.loadv2.BrokerLoadJobTest - Behavior changed: No. This completes the consistency and persistence guarantees of the recovery path added by the PR. - Does this need documentation: No

hello-stephen · 2026-08-21T04:47:54Z

ClickBench: Total hot run time: 14.71 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit c3967b9b3b1640ae2a2107575fa7bcbc26c068b0, data reload: false
query1	0.00	0.00	0.00
query2	0.07	0.03	0.04
query3	0.24	0.10	0.10
query4	1.60	0.10	0.10
query5	0.18	0.16	0.15
query6	1.24	0.72	0.71
query7	0.03	0.00	0.01
query8	0.04	0.03	0.04
query9	0.29	0.21	0.21
query10	0.37	0.34	0.35
query11	0.16	0.11	0.12
query12	0.14	0.11	0.12
query13	0.30	0.30	0.30
query14	0.48	0.45	0.44
query15	0.36	0.35	0.34
query16	0.21	0.23	0.22
query17	0.67	0.68	0.68
query18	0.18	0.17	0.18
query19	1.20	1.20	1.24
query20	0.01	0.01	0.01
query21	15.46	0.17	0.13
query22	5.05	0.04	0.04
query23	16.14	0.26	0.10
query24	3.10	0.31	0.26
query25	0.11	0.05	0.04
query26	0.74	0.16	0.12
query27	0.04	0.03	0.03
query28	3.69	0.55	0.27
query29	12.59	3.13	2.53
query30	0.28	0.12	0.12
query31	2.76	0.38	0.17
query32	3.51	0.32	0.23
query33	1.43	1.46	1.43
query34	15.37	2.11	1.75
query35	1.77	1.71	1.75
query36	0.46	0.29	0.28
query37	0.06	0.04	0.04
query38	0.04	0.04	0.04
query39	0.03	0.02	0.02
query40	0.11	0.08	0.08
query41	0.08	0.03	0.03
query42	0.04	0.02	0.02
query43	0.04	0.02	0.03
Total cold run time: 90.67 s
Total hot run time: 14.71 s

hello-stephen · 2026-08-21T08:29:50Z

FE Regression Coverage Report

Increment line coverage 0.00% (0/88) 🎉
Increment coverage report
Complete coverage report

liaoxin01

LGTM

done

### What problem does this PR solve? In cloud mode, a Broker Load transaction can become VISIBLE in Meta Service before FE persists the final Load Job record. If FE restarts in that window, the replayed PENDING job begins the same label again and is cancelled after conflicting with its own visible transaction. This change looks up the transaction by label, verifies that its callback ID belongs to the current Load Job, reuses PREPARE transactions, and finishes the job immediately when the transaction is already VISIBLE. For Cloud Broker Load, the final operation is persisted to EditLog. ### Release note Fix Broker Load jobs being reported as cancelled after FE restart even though their transactions were already visible.

…66987 (#67075) Cherry-picked from #66987 Co-authored-by: hui lai <laihui@selectdb.com>

Related PR: #66987 Problem Summary: In cloud mode, a Broker Load attempt can have many loading tasks queued in the FE executor. After the first task failure moves the job to RETRY, a queued task can currently move it back to LOADING. This allows several concurrent failures to enter the job-level retry path and wait on the shared idToTasks map. An older retry handler can consequently follow tasks from a newer attempt. If that newer attempt finishes successfully, the stale handler can overwrite FINISHED with PENDING, schedule another pending task, and reuse the already VISIBLE transaction. New rowsets are then rejected by Meta Service because the transaction is no longer PREPARED. This change closes the current attempt once the job enters RETRY and verifies that the job is still RETRY before scheduling the next PENDING attempt. It does not change transaction protocols, Meta Service behavior, or the non-cloud retry implementation. ### Release note Fix Cloud Broker Load jobs being retried after a later attempt already finished.

sollhui marked this pull request as ready for review August 21, 2026 02:45

sollhui requested review from gavinchou and liaoxin01 as code owners August 21, 2026 02:45

sollhui changed the title ~~[fix](load) Recover broker load from visible transaction~~[fix](job) Recover broker load from visible transactionAug 21, 2026

github-actionsBot previously requested changes Aug 21, 2026
View reviewed changes

sollhui force-pushed the fix/broker-load-visible-txn-recovery branch from 9cf58df to c3967b9Compare August 21, 2026 04:55

sollhui added dev/4.0.x dev/4.1.x labels Aug 21, 2026

gavinchou approved these changes Aug 24, 2026
View reviewed changes

liaoxin01 approved these changes Aug 24, 2026
View reviewed changes

liaoxin01 merged commit 5f72a92 into apache:masterAug 24, 2026
56 checks passed

github-actionsBot mentioned this pull request Aug 24, 2026
branch-4.1: [fix](job) Recover broker load from visible transaction #66987 #67075
Merged

yiguolei pushed a commit that referenced this pull request Aug 25, 2026
branch-4.1: [fix](job) Recover broker load from visible transaction#…
2556448
…66987 (#67075) Cherry-picked from #66987 Co-authored-by: hui lai <laihui@selectdb.com>

yiguolei added dev/4.1.4-merged and removed dev/4.1.x labels Aug 25, 2026

0AyanamiRei mentioned this pull request Aug 28, 2026
[fix](job) Prevent stale cloud broker load retries #67250
Merged
12 tasks

yiguolei mentioned this pull request Sep 1, 2026
4.1.4 Release Note #67355
Open

Navigation Menu

Uh oh!

[fix](job) Recover broker load from visible transaction - #66987

[fix](job) Recover broker load from visible transaction#66987
liaoxin01 merged 1 commit into
apache:masterfrom
sollhui:fix/broker-load-visible-txn-recovery

sollhui commented Aug 20, 2026•
edited
Loading

Uh oh!

hello-stephen commented Aug 20, 2026

Uh oh!

sollhui commented Aug 21, 2026

Uh oh!

sollhui commented Aug 21, 2026

Uh oh!

github-actionsBot left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

hello-stephen commented Aug 21, 2026

Uh oh!

hello-stephen commented Aug 21, 2026

Uh oh!

hello-stephen commented Aug 21, 2026

Uh oh!

hello-stephen commented Aug 21, 2026

Uh oh!

hello-stephen commented Aug 21, 2026

Uh oh!

liaoxin01 left a comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Uh oh!

Conversation

sollhui commented Aug 20, 2026• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What problem does this PR solve?

Release note

Check List (For Author)

Check List (For Reviewer who merge this PR)

Uh oh!

hello-stephen commented Aug 20, 2026

Uh oh!

sollhui commented Aug 21, 2026

Uh oh!

sollhui commented Aug 21, 2026

Uh oh!

github-actionsBot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

hello-stephen commented Aug 21, 2026

FE UT Coverage Report

Uh oh!

hello-stephen commented Aug 21, 2026

Uh oh!

hello-stephen commented Aug 21, 2026

Uh oh!

hello-stephen commented Aug 21, 2026

Uh oh!

hello-stephen commented Aug 21, 2026

FE Regression Coverage Report

Uh oh!

liaoxin01 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

sollhui commented Aug 20, 2026•
edited
Loading