[fix](be) Synchronize load error log access by 0AyanamiRei · Pull Request #66250 · apache/doris

0AyanamiRei · 2026-07-30T02:42:04Z

What problem does this PR solve?

Issue Number: None

Related PR: #41320

Load error-log writing and periodic status reporting can concurrently access the same std::ofstream. The reporting path may close and upload the stream while the writer is appending an error row, causing a data race and a BE SIGSEGV in the stream write path.

This PR uses _load_error_log_lock to serialize stream creation and writes and to protect the first-error and path state. A separate _s3_error_log_file_lock serializes remote upload and URL publication, while synchronous S3 I/O runs without holding the load-error writer lock.

Release note

None

Check List (For Author)

Test: Local build
Behavior changed: No. This only synchronizes existing load error-log behavior.
Does this need documentation: No

### What problem does this PR solve? Issue Number: None Related PR: apache#41320 Problem Summary: Load error-log writing and periodic status reporting can concurrently access the same std::ofstream. The reporting path may close and upload the stream while the writer is appending an error row, causing a data race and a BE SIGSEGV in the stream write path. Protect error-log creation, first-error state, stream writes, path retrieval, close, upload, and URL generation with one mutex. Add a fault-injection regression that holds the writer before its first write while periodic reporting requests the error-log path. ### Release note None ### Check List (For Author) - Test: Regression test and local build - BUILD_TYPE=Debug ./build.sh --be -j48 - BUILD_TYPE=Debug ./build.sh --fe --be --cloud -j48 - ./run-regression-test.sh --run -d fault_injection_p0/cloud -s test_load_error_log_concurrent_write -times 3 - Behavior changed: No. This only synchronizes existing load error-log behavior. - Does this need documentation: No

hello-stephen · 2026-07-30T02:42:08Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

What problem was fixed (it's best to include specific error reporting information). How it was fixed.
Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
What features were added. Why was this function added?
Which code was refactored and why was this part of the code refactored?
Which functions were optimized and what is the difference before and after the optimization?

sollhui · 2026-07-30T04:47:05Z

/review

github-actions

Request changes.

The mutex addresses the immediate unsynchronized std::ofstream access, but the end-to-end writer/reporter lifecycle still has three blockers covered inline: periodic upload leaves the active writer closed and silently drops later rows; synchronous S3 transfer now blocks the async writer under the mutex; and the regression does not deterministically prove reporter/writer overlap.

Checkpoint conclusions:

Goal and proof: the change removes simultaneous stream access, but it does not preserve complete error-log contents across periodic reporting, and the timing-only test can pass without exercising the old race.
Scope: the diff is small and focused, but the synchronization remedy is incomplete because stream finalization, local-path/URL state, and upload snapshot ownership are not modeled explicitly.
Concurrency and locks: the competing contexts are the FragmentMgr async result-writer worker and pipeline task-scheduler reporting worker. Shared RuntimeState fields are now guarded and no lock inversion, callback re-entry, or permanent deadlock was found, but remote I/O inside the critical section creates a new writer/backpressure stall.
Lifecycle, ownership, memory, and errors: _error_log_file remains non-null after close, later ofstream failures are not checked, and _error_log_file_path changes meaning from local path to URL. No new ownership cycle, untracked allocation, cache-handle issue, or static-initialization hazard was found.
Configuration and compatibility: no new config, FE/BE variable, protocol, storage format, persistence, or rolling-upgrade behavior is introduced. The test uses existing mutable BE configs.
Parallel paths: all append, path, and first-message callers were traced; the issue is common to load sink paths using the per-task RuntimeState, not an isolated caller omission.
Testing: the cloud regression follows table setup/cleanup conventions, but lacks a two-party synchronization point and post-upload append coverage. Per the review-only runner contract, no build or test was executed during this review.
Observability, transactions, and data semantics: this change affects diagnostic error-log reporting only; no transaction, visible-version, delete-bitmap, or committed-data path is modified. Existing identifiers/logging are otherwise adequate.
Performance: S3 upload reads and transfers the whole error file synchronously while holding the new mutex; with the configured limits this is not a bounded lightweight critical section.
User focus: no additional user-provided focus was supplied; the complete PR was reviewed.

0AyanamiRei · 2026-08-03T01:08:01Z

run buildall

hello-stephen · 2026-08-03T04:14:47Z

BE UT Coverage Report

Increment line coverage 44.74% (17/38) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	59.16% (25632/43323)
Line Coverage	43.27% (257411/594874)
Region Coverage	38.94% (203966/523795)
Branch Coverage	40.29% (93076/231041)

hello-stephen · 2026-08-03T05:32:59Z

BE Regression && UT Coverage Report

Increment line coverage 89.47% (34/38) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	75.74% (32005/42254)
Line Coverage	60.41% (356956/590898)
Region Coverage	57.00% (299561/525562)
Branch Coverage	58.41% (134963/231079)

sollhui · 2026-08-03T06:24:13Z

/review

hello-stephen · 2026-08-03T06:30:49Z

TPC-H: Total hot run time: 28717 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 6ddee4cd1e4ecb8a7d06ed5b72a01a9749d07278, data reload: false
------ Round 1 ----------------------------------
============================================
q1	17859	4027	3995	3995
q2	2030	320	207	207
q3	10284	1403	809	809
q4	4713	471	338	338
q5	7649	845	554	554
q6	226	174	136	136
q7	771	796	595	595
q8	10493	1552	1579	1552
q9	5631	4039	3986	3986
q10	6832	1645	1353	1353
q11	503	363	345	345
q12	768	597	447	447
q13	18078	3365	2756	2756
q14	264	258	232	232
q15	q16	738	730	668	668
q17	1022	900	1010	900
q18	7027	5697	5554	5554
q19	1181	1191	1022	1022
q20	833	677	582	582
q21	5512	2518	2386	2386
q22	446	368	300	300
Total cold run time: 102860 ms
Total hot run time: 28717 ms
----- Round 2, with runtime_filter_mode=off -----
============================================
q1	4430	4259	4263	4259
q2	283	304	210	210
q3	4541	4959	4462	4462
q4	2111	2268	1413	1413
q5	4228	4105	4112	4105
q6	224	171	129	129
q7	1739	1588	1901	1588
q8	2546	2191	2132	2132
q9	7339	7188	7146	7146
q10	4345	4347	3912	3912
q11	551	420	362	362
q12	748	714	503	503
q13	3241	3469	3045	3045
q14	295	296	290	290
q15	q16	701	732	644	644
q17	1360	1302	1313	1302
q18	8012	7340	6988	6988
q19	1085	1037	1041	1037
q20	2204	2211	1922	1922
q21	5281	4558	4452	4452
q22	544	498	414	414
Total cold run time: 55808 ms
Total hot run time: 50315 ms

hello-stephen · 2026-08-03T06:41:38Z

TPC-DS: Total hot run time: 169923 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 6ddee4cd1e4ecb8a7d06ed5b72a01a9749d07278, data reload: false
query5	4313	615	475	475
query6	465	221	203	203
query7	4903	571	350	350
query8	332	181	162	162
query9	8744	4031	4074	4031
query10	519	344	297	297
query11	5821	2205	2006	2006
query12	150	100	97	97
query13	1260	590	418	418
query14	6164	4868	4411	4411
query14_1	3838	3856	3779	3779
query15	209	196	176	176
query16	1028	466	415	415
query17	1097	662	530	530
query18	2459	446	325	325
query19	197	185	138	138
query20	104	98	98	98
query21	231	156	149	149
query22	13097	13036	12911	12911
query23	17372	16390	16132	16132
query23_1	16155	16164	16107	16107
query24	7579	1665	1241	1241
query24_1	1293	1248	1236	1236
query25	566	455	388	388
query26	1345	363	220	220
query27	2626	590	382	382
query28	4457	2032	2033	2032
query29	1074	624	492	492
query30	347	264	226	226
query31	1108	1081	970	970
query32	106	64	64	64
query33	532	310	261	261
query34	1169	1111	632	632
query35	736	771	645	645
query36	790	759	658	658
query37	153	107	95	95
query38	1827	1663	1612	1612
query39	823	829	805	805
query39_1	794	780	780	780
query40	250	161	148	148
query41	73	71	75	71
query42	98	94	93	93
query43	315	314	282	282
query44	1449	781	766	766
query45	188	181	172	172
query46	1025	1200	729	729
query47	1528	1516	1461	1461
query48	401	431	294	294
query49	598	421	302	302
query50	1046	435	340	340
query51	10355	10415	10784	10415
query52	88	92	77	77
query53	278	278	199	199
query54	305	247	225	225
query55	75	73	67	67
query56	322	288	286	286
query57	1019	994	930	930
query58	288	273	248	248
query59	1501	1592	1426	1426
query60	318	279	258	258
query61	181	171	171	171
query62	404	325	272	272
query63	241	196	200	196
query64	2931	1026	836	836
query65	3932	3857	3837	3837
query66	1848	460	379	379
query67	28470	28225	28100	28100
query68	3095	1567	1011	1011
query69	413	296	268	268
query70	866	787	773	773
query71	369	325	323	323
query72	3076	2669	2285	2285
query73	799	769	445	445
query74	4629	4494	4293	4293
query75	2382	2353	2003	2003
query76	2309	1131	764	764
query77	345	374	275	275
query78	11242	11137	10700	10700
query79	1348	1127	764	764
query80	1194	545	462	462
query81	499	332	283	283
query82	590	158	110	110
query83	389	319	300	300
query84	287	158	131	131
query85	958	596	512	512
query86	364	240	223	223
query87	1829	1794	1699	1699
query88	3706	2808	2796	2796
query89	400	316	287	287
query90	1832	201	198	198
query91	202	188	161	161
query92	60	59	53	53
query93	1570	1553	976	976
query94	625	352	317	317
query95	775	590	473	473
query96	1092	853	366	366
query97	2456	2477	2333	2333
query98	206	197	188	188
query99	734	721	608	608
Total cold run time: 256115 ms
Total hot run time: 169923 ms

hello-stephen · 2026-08-03T06:46:35Z

ClickBench: Total hot run time: 23.87 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 6ddee4cd1e4ecb8a7d06ed5b72a01a9749d07278, data reload: false
query1	0.01	0.00	0.01
query2	0.09	0.05	0.05
query3	0.26	0.14	0.14
query4	1.61	0.14	0.14
query5	0.24	0.23	0.23
query6	1.16	0.84	0.82
query7	0.04	0.00	0.00
query8	0.06	0.03	0.04
query9	0.39	0.32	0.33
query10	0.56	0.59	0.54
query11	0.19	0.14	0.14
query12	0.18	0.14	0.15
query13	0.46	0.46	0.47
query14	1.01	1.01	0.97
query15	0.61	0.58	0.59
query16	0.35	0.34	0.32
query17	1.11	1.16	1.07
query18	0.23	0.21	0.21
query19	2.10	1.97	2.00
query20	0.01	0.01	0.02
query21	15.43	0.20	0.14
query22	4.92	0.06	0.05
query23	16.13	0.30	0.12
query24	2.93	0.44	0.32
query25	0.10	0.05	0.04
query26	0.72	0.21	0.15
query27	0.04	0.04	0.03
query28	3.51	0.78	0.37
query29	12.50	4.04	3.17
query30	0.27	0.14	0.15
query31	2.77	0.56	0.32
query32	3.22	0.60	0.49
query33	3.24	3.27	3.13
query34	15.41	3.99	3.29
query35	3.22	3.20	3.25
query36	0.56	0.43	0.42
query37	0.09	0.07	0.06
query38	0.06	0.04	0.03
query39	0.04	0.03	0.02
query40	0.18	0.16	0.15
query41	0.08	0.03	0.03
query42	0.04	0.03	0.03
query43	0.04	0.03	0.03
Total cold run time: 96.17 s
Total hot run time: 23.87 s

github-actions

Current-head follow-up: no new unique inline finding remains after full review and deduplication on 6ddee4cd1e4ecb8a7d06ed5b72a01a9749d07278.

The current head addresses the prior remote-I/O critical-section concern: S3 upload and presigning now run after _load_error_log_lock is released, so that thread's writer-blocking premise no longer applies to this revision. Two existing blockers remain and are not duplicated here: periodic/cancellation reporting still closes the non-null active writer and silently loses later rows, and the timing-only regression still does not prove reporter/writer overlap or fail on the unfixed path. Those unresolved threads still prevent approval.

Checkpoint conclusions:

Goal and proof: the mutex removes simultaneous ofstream close/write access and protects first-error reads, but the end-to-end error-log lifecycle remains incomplete because later writes can target the closed stream. The added test is not a deterministic proof of the fix.
Scope and clarity: the three-file diff is small and focused, but a correct fix still needs an explicit active/finalized or rotating-snapshot state rather than treating close/upload as a terminal transition during periodic reporting.
Concurrency and locks: scanner/format writer tasks compete with pipeline periodic/cancellation reporting over the per-RuntimeState stream and path state. The internal nested order is consistently _s3_error_log_file_lock then _load_error_log_lock; writers/readers take only the load lock, so no reverse edge or distinct deadlock was found. Concurrent getters are serialized through close/upload/publish.
Lifecycle, ownership, and memory: the copied S3 filesystem handle safely owns the upload dependency and no new ownership cycle, cache-handle leak, untracked long-lived buffer, thread-context issue, or static-initialization hazard was found. The remaining non-null-but-closed stream state is the existing lifecycle blocker.
Configuration: no configuration item is added. The regression uses existing mutable BE configs before fragment initialization.
Compatibility: no function/protocol symbol, storage format, FE/BE transport, persistence, or rolling-upgrade contract changes.
Parallel paths: periodic, cancellation, normal final, stream-load, group-commit, internal-service, and multi-table paths were traced. Periodic/cancellation paths reproduce the existing closed-writer issue; normal final callbacks run after pipeline tasks close. Repeated getter calls are redundant but stable.
Conditions and error handling: upload Status is checked and logged, and first-error access is synchronized. Closed-stream writes still return OK without checking stream state, already covered by the existing lifecycle thread.
Testing and result correctness: the cloud test follows drop-before-use and cleanup conventions, but its sleep has no two-party handshake, the debug point is absent on the base revision, and one invalid row cannot prove post-report durability. No generated result file is involved. Per the review-runner contract, no build or test was executed.
Observability: existing creation/upload warnings and query/fragment identifiers are adequate for this scoped change; no distinct missing log or metric was substantiated.
Transactions, persistence, and data semantics: this changes diagnostic error-log handling only; no EditLog, master-failover, visible-version, delete-bitmap, or committed-table-data path is modified. Error-log upload completeness remains governed by the existing lifecycle blocker.
Performance: remote S3 I/O is now outside _load_error_log_lock. Formatting and local stream output remain serialized, but no separate correctness or critical-executor failure beyond the existing rotation/snapshot discussion was substantiated.
Additional review: no other correctness, compatibility, memory, observability, or performance issue survived concrete call-chain verification and duplicate suppression.
User focus: no additional user-provided focus was supplied; the complete PR was reviewed.

### What problem does this PR solve? Issue Number: None Related PR: apache#41320 Problem Summary: Remove the fault-injection regression case and the debug point used only by that case. Keep the production load error-log synchronization change unchanged. ### Release note None ### Check List (For Author) - Test: No need to test (only removes a regression case and its test-only debug point; run buildall will validate the updated PR head) - Behavior changed: No - Does this need documentation: No

0AyanamiRei · 2026-08-04T02:22:32Z

run buildall

hello-stephen · 2026-08-04T04:28:58Z

TPC-H: Total hot run time: 29038 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit fc852ae9b2f2c1e580bfaabddc5affb7a7797e0b, data reload: false
------ Round 1 ----------------------------------
============================================
q1	17735	4092	4009	4009
q2	2030	338	208	208
q3	10288	1472	795	795
q4	4678	476	342	342
q5	7515	855	551	551
q6	184	172	142	142
q7	757	806	608	608
q8	9370	1561	1572	1561
q9	5404	4102	4099	4099
q10	6752	1638	1368	1368
q11	505	351	341	341
q12	710	580	461	461
q13	18116	3363	2757	2757
q14	268	255	249	249
q15	q16	736	742	660	660
q17	1021	1000	919	919
q18	6642	5615	5558	5558
q19	1369	1187	1115	1115
q20	830	668	600	600
q21	5950	2690	2393	2393
q22	443	352	302	302
Total cold run time: 101303 ms
Total hot run time: 29038 ms
----- Round 2, with runtime_filter_mode=off -----
============================================
q1	4339	4323	4245	4245
q2	293	325	207	207
q3	4581	4971	4425	4425
q4	2170	2251	1417	1417
q5	4244	4092	4128	4092
q6	227	177	133	133
q7	1689	1573	1455	1455
q8	2848	2225	2102	2102
q9	7446	7519	7540	7519
q10	4367	4285	3935	3935
q11	552	409	359	359
q12	713	726	507	507
q13	3224	3502	2943	2943
q14	307	307	274	274
q15	q16	695	707	642	642
q17	1316	1277	1332	1277
q18	12151	11075	11808	11075
q19	1205	1137	1149	1137
q20	2236	2210	1930	1930
q21	5709	4892	4838	4838
q22	541	483	425	425
Total cold run time: 60853 ms
Total hot run time: 54937 ms

hello-stephen · 2026-08-04T04:39:37Z

TPC-DS: Total hot run time: 166681 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit fc852ae9b2f2c1e580bfaabddc5affb7a7797e0b, data reload: false
query5	4348	578	456	456
query6	469	219	213	213
query7	4867	601	349	349
query8	326	161	150	150
query9	8745	4084	4053	4053
query10	514	346	304	304
query11	5846	2174	1983	1983
query12	157	101	97	97
query13	1243	584	421	421
query14	6140	4276	3980	3980
query14_1	3811	3818	3801	3801
query15	199	194	183	183
query16	1026	497	470	470
query17	934	724	571	571
query18	2442	496	368	368
query19	203	180	143	143
query20	105	99	102	99
query21	230	154	135	135
query22	13144	13020	12829	12829
query23	15791	14974	14566	14566
query23_1	14624	14630	14736	14630
query24	7528	1712	1237	1237
query24_1	1206	1244	1236	1236
query25	530	453	362	362
query26	1322	340	212	212
query27	2640	588	398	398
query28	4553	2056	2050	2050
query29	1042	604	462	462
query30	344	261	223	223
query31	1172	1152	1057	1057
query32	106	61	60	60
query33	511	297	234	234
query34	1214	1148	620	620
query35	734	752	626	626
query36	771	782	712	712
query37	153	109	92	92
query38	1837	1759	1689	1689
query39	834	821	813	813
query39_1	793	785	801	785
query40	251	166	143	143
query41	66	63	67	63
query42	94	94	93	93
query43	314	318	282	282
query44	1416	788	778	778
query45	186	173	171	171
query46	1029	1214	695	695
query47	1536	1508	1472	1472
query48	415	412	301	301
query49	564	414	303	303
query50	1084	430	344	344
query51	10589	10288	10604	10288
query52	90	85	73	73
query53	264	265	203	203
query54	289	226	220	220
query55	74	72	74	72
query56	336	299	301	299
query57	1003	988	953	953
query58	276	248	250	248
query59	1546	1623	1425	1425
query60	303	288	276	276
query61	186	171	173	171
query62	411	319	275	275
query63	238	200	207	200
query64	3031	1184	1011	1011
query65	3897	3834	3852	3834
query66	1873	496	373	373
query67	28202	28220	27973	27973
query68	3185	1599	1085	1085
query69	420	310	267	267
query70	867	815	785	785
query71	367	350	325	325
query72	3181	2618	2300	2300
query73	850	781	453	453
query74	4610	4526	4299	4299
query75	2364	2349	2009	2009
query76	2330	1132	771	771
query77	343	361	271	271
query78	11290	11338	10561	10561
query79	1406	1138	758	758
query80	693	571	471	471
query81	482	328	282	282
query82	633	174	140	140
query83	411	330	304	304
query84	323	155	128	128
query85	939	618	536	536
query86	318	227	236	227
query87	1989	1954	1841	1841
query88	3717	2800	2807	2800
query89	401	326	283	283
query90	1929	209	192	192
query91	202	191	163	163
query92	63	56	55	55
query93	1539	1538	1055	1055
query94	568	364	291	291
query95	801	505	459	459
query96	1082	762	391	391
query97	2481	2465	2331	2331
query98	199	186	191	186
query99	747	718	609	609
Total cold run time: 252830 ms
Total hot run time: 166681 ms

hello-stephen · 2026-08-04T04:44:33Z

ClickBench: Total hot run time: 23.83 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit fc852ae9b2f2c1e580bfaabddc5affb7a7797e0b, data reload: false
query1	0.01	0.01	0.01
query2	0.09	0.05	0.05
query3	0.25	0.13	0.13
query4	1.60	0.14	0.13
query5	0.24	0.23	0.22
query6	1.16	0.79	0.82
query7	0.04	0.01	0.01
query8	0.06	0.04	0.04
query9	0.37	0.31	0.31
query10	0.55	0.60	0.53
query11	0.19	0.14	0.14
query12	0.18	0.14	0.14
query13	0.46	0.48	0.48
query14	1.00	0.99	1.00
query15	0.60	0.59	0.59
query16	0.31	0.31	0.31
query17	1.07	1.12	1.06
query18	0.22	0.20	0.20
query19	2.07	1.90	1.96
query20	0.02	0.01	0.02
query21	15.42	0.23	0.15
query22	4.93	0.05	0.05
query23	16.15	0.31	0.12
query24	2.90	0.44	0.32
query25	0.12	0.05	0.05
query26	0.75	0.20	0.14
query27	0.04	0.04	0.05
query28	3.51	0.76	0.35
query29	12.53	4.06	3.16
query30	0.28	0.15	0.17
query31	2.77	0.55	0.31
query32	3.21	0.58	0.49
query33	3.31	3.16	3.18
query34	15.51	3.96	3.32
query35	3.25	3.21	3.22
query36	0.56	0.42	0.42
query37	0.08	0.06	0.06
query38	0.04	0.03	0.04
query39	0.04	0.04	0.03
query40	0.17	0.16	0.14
query41	0.08	0.03	0.03
query42	0.04	0.03	0.02
query43	0.04	0.04	0.04
Total cold run time: 96.22 s
Total hot run time: 23.83 s

hello-stephen · 2026-08-04T07:04:54Z

BE UT Coverage Report

Increment line coverage 48.57% (17/35) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	59.30% (25736/43401)
Line Coverage	43.37% (258442/595914)
Region Coverage	39.08% (205035/524695)
Branch Coverage	40.40% (93514/231481)

hello-stephen · 2026-08-04T07:08:48Z

BE Regression && UT Coverage Report

Increment line coverage 88.57% (31/35) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	75.85% (32101/42322)
Line Coverage	60.53% (358236/591878)
Region Coverage	57.07% (300430/526440)
Branch Coverage	58.51% (135456/231505)

sollhui · 2026-08-13T09:43:46Z

/review

github-actions

Request changes.

The current head fixes the prior remote-I/O critical section: S3 upload and presigning now run outside _load_error_log_lock, so that earlier writer-blocking concern no longer applies. Two existing blockers remain and are not duplicated inline here: periodic/cancellation reporting still closes the non-null active writer and silently loses later rows, and there is still no deterministic test proving writer/reporter overlap or failing when the synchronization fix is reverted. This review adds one distinct inline performance finding: all invalid rows after the bounded log reaches its 51-row cap still serialize on the new writer mutex.

Checkpoint conclusions:

Goal and proof: the mutex removes simultaneous ofstream close/write access and synchronizes the first-error read, but it does not preserve complete error-log contents when periodic or cancellation reporting runs before writers finish. The authoritative diff contains no deterministic regression proof.
Scope and clarity: the two-file diff is small and focused, but the design still conflates an active local writer, a closed upload snapshot, and the published URL instead of modeling the lifecycle explicitly.
Concurrency and locks: parallel scanner-pool and pipeline workers append while pipeline scheduling/reporting and cancellation paths request the URL. Lock order is consistently _s3_error_log_file_lock then _load_error_log_lock; no reverse acquisition or distinct deadlock was found. The new mutex is nevertheless on the per-rejected-row hot path even after no further log write is possible.
Lifecycle, ownership, memory, and errors: the copied S3 filesystem handle safely owns the upload dependency, and no ownership cycle, cache-handle leak, untracked long-lived allocation, thread-context issue, or static-initialization hazard was found. The non-null-but-closed stream still accepts failed insertions without checking stream state and returns OK, as covered by the existing lifecycle thread. Upload Status is checked and logged.
Configuration and compatibility: no configuration item, FE/BE variable, protocol, storage format, persistence schema, or rolling-upgrade contract is added or changed.
Parallel and failure paths: periodic, cancellation, ordinary final, stream-load, group-commit, internal-service, multi-table, repeated-getter, upload-success, and upload-failure paths were traced. Normal final reporting has the task-completion fence; periodic/cancellation paths do not. No additional distinct failure survived duplicate suppression.
Testing and results: no test or generated result file is changed. The existing fault-injection test uses only a sleep, without a two-party handshake or revert-failure assertion. Per the review-only runner contract, no build or test was executed.
Observability, transactions, and data semantics: existing creation/upload logging has adequate identifiers for this scoped path. No transaction, EditLog, master-failover, visible-version, delete-bitmap, or committed-table-data behavior is modified; the affected data is diagnostic error-log content.
Performance: remote S3 I/O is now outside the writer mutex. A high-filter-ratio file can still funnel concurrent scanners through that mutex for every invalid row after the 51-row logging cap, which is the new inline issue.
Additional review and user focus: all changed lines and callers were swept after two full normal reviews and two risk-focused passes; all subagents converged with no further valuable findings. No additional user-provided focus was supplied, and the complete PR was reviewed.

### What problem does this PR solve? Issue Number: None Related PR: apache#41320 Problem Summary: Once load error logging reaches MAX_ERROR_NUM, concurrent scanner workers still acquire _load_error_log_lock before the counter rejects later rows. Add a relaxed atomic pre-check before the mutex and retain the locked fetch_add as the authoritative recheck and slot claim. The fast path preserves the existing error-row limit and zero-tolerance result while avoiding post-cap writer serialization. ### Release note None ### Check List (For Author) - Test: No need to test (no functional behavior change; compilation will be covered by PR buildall) - Behavior changed: No - Does this need documentation: No

0AyanamiRei · 2026-08-16T16:47:40Z

run buildall

hello-stephen · 2026-08-16T18:50:11Z

BE UT Coverage Report

Increment line coverage 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	60.19% (26591/44182)
Line Coverage	44.70% (271328/607058)
Region Coverage	40.37% (216371/535943)
Branch Coverage	41.92% (99554/237474)

hello-stephen · 2026-08-16T18:58:01Z

TPC-H: Total hot run time: 17734 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 3beb646cf26fc78b499799ba16bb3b5dd050be8c, data reload: false
------ Round 1 ----------------------------------
============================================
q1	17620	3159	3137	3137
q2	2094	230	151	151
q3	10270	874	507	507
q4	4665	247	199	199
q5	7682	557	384	384
q6	137	115	93	93
q7	527	504	387	387
q8	9256	871	875	871
q9	3451	2350	2352	2350
q10	6515	838	676	676
q11	467	262	236	236
q12	689	391	337	337
q13	17944	2098	1528	1528
q14	161	145	135	135
q15	q16	444	403	362	362
q17	851	849	758	758
q18	3036	2236	2230	2230
q19	1303	913	795	795
q20	614	519	447	447
q21	5632	1929	1955	1929
q22	318	266	222	222
Total cold run time: 93676 ms
Total hot run time: 17734 ms
----- Round 2, with runtime_filter_mode=off -----
============================================
q1	3523	3497	3428	3428
q2	209	210	153	153
q3	2200	2303	2084	2084
q4	1194	1161	894	894
q5	2202	2135	2092	2092
q6	173	130	88	88
q7	1033	924	891	891
q8	1621	1414	1422	1414
q9	3087	3078	3101	3078
q10	1837	1790	1597	1597
q11	364	268	250	250
q12	471	449	344	344
q13	1815	1852	1548	1548
q14	174	167	171	167
q15	q16	391	392	362	362
q17	1057	1045	1045	1045
q18	4965	4339	4762	4339
q19	889	847	839	839
q20	956	955	937	937
q21	3592	3028	3248	3028
q22	389	355	318	318
Total cold run time: 32142 ms
Total hot run time: 28896 ms

hello-stephen · 2026-08-16T19:04:45Z

TPC-DS: Total hot run time: 84587 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 3beb646cf26fc78b499799ba16bb3b5dd050be8c, data reload: false
query5	4222	401	328	328
query6	417	160	146	146
query7	4906	469	270	270
query8	303	122	121	121
query9	8697	2905	2895	2895
query10	458	264	213	213
query11	5409	1028	902	902
query12	117	70	70	70
query13	1200	447	344	344
query14	5963	2003	1865	1865
query14_1	1785	1760	1784	1760
query15	178	120	111	111
query16	3030	386	379	379
query17	798	467	377	377
query18	2111	342	232	232
query19	176	138	108	108
query20	70	68	67	67
query21	414	113	103	103
query22	5345	5265	5243	5243
query23	7304	6552	6446	6446
query23_1	6400	6618	6535	6535
query24	7311	1046	779	779
query24_1	762	756	781	756
query25	423	285	256	256
query26	1237	276	159	159
query27	2671	461	277	277
query28	4639	1488	1491	1488
query29	922	430	332	332
query30	410	179	151	151
query31	981	664	584	584
query32	100	48	47	47
query33	486	199	170	170
query34	1020	829	473	473
query35	400	387	341	341
query36	553	546	513	513
query37	126	84	70	70
query38	996	834	802	802
query39	539	540	528	528
query39_1	508	508	524	508
query40	229	121	105	105
query41	54	52	52	52
query42	73	71	77	71
query43	247	244	221	221
query44	1032	563	567	563
query45	120	110	100	100
query46	788	884	543	543
query47	969	938	942	938
query48	327	299	240	240
query49	541	249	201	201
query50	836	388	291	291
query51	8051	7979	7898	7898
query52	69	70	62	62
query53	238	213	151	151
query54	229	185	175	175
query55	72	56	53	53
query56	258	206	204	204
query57	764	609	611	609
query58	241	190	179	179
query59	1077	1067	992	992
query60	281	198	188	188
query61	113	108	115	108
query62	360	210	173	173
query63	191	154	150	150
query64	2500	665	608	608
query65	1542	1574	1576	1574
query66	1896	305	281	281
query67	9880	9434	9547	9434
query68	2742	1260	811	811
query69	503	218	195	195
query70	657	611	606	606
query71	303	251	229	229
query72	2363	1395	1562	1395
query73	648	570	351	351
query74	1562	1227	1127	1127
query75	1228	1127	988	988
query76	1870	739	536	536
query77	240	246	201	201
query78	5108	4724	4461	4461
query79	2406	842	588	588
query80	1512	383	336	336
query81	512	190	167	167
query82	680	143	112	112
query83	309	255	238	238
query84	310	128	101	101
query85	866	431	380	380
query86	418	170	177	170
query87	1001	977	889	889
query88	2808	2120	2110	2110
query89	320	229	211	211
query90	1890	146	147	146
query91	157	145	120	120
query92	59	49	42	42
query93	1490	1165	788	788
query94	666	246	228	228
query95	660	372	372	372
query96	831	568	274	274
query97	1087	1050	1008	1008
query98	149	140	127	127
query99	495	342	304	304
Total cold run time: 181788 ms
Total hot run time: 84587 ms

hello-stephen · 2026-08-16T19:09:09Z

ClickBench: Total hot run time: 14.61 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 3beb646cf26fc78b499799ba16bb3b5dd050be8c, data reload: false
query1	0.01	0.00	0.01
query2	0.19	0.04	0.04
query3	0.28	0.13	0.11
query4	1.61	0.09	0.10
query5	0.17	0.16	0.16
query6	1.24	0.67	0.69
query7	0.03	0.01	0.00
query8	0.05	0.03	0.03
query9	0.36	0.24	0.24
query10	0.37	0.34	0.34
query11	0.22	0.11	0.12
query12	0.20	0.12	0.11
query13	0.31	0.32	0.30
query14	0.48	0.45	0.48
query15	0.36	0.34	0.34
query16	0.22	0.23	0.23
query17	0.68	0.69	0.67
query18	0.18	0.16	0.17
query19	1.11	1.12	1.16
query20	0.01	0.01	0.01
query21	15.55	0.15	0.11
query22	4.99	0.05	0.04
query23	16.36	0.25	0.10
query24	4.83	0.33	0.30
query25	0.13	0.04	0.02
query26	0.72	0.15	0.12
query27	0.05	0.03	0.03
query28	3.62	0.52	0.28
query29	12.42	3.18	2.55
query30	0.27	0.12	0.11
query31	2.76	0.36	0.17
query32	3.52	0.31	0.23
query33	1.36	1.41	1.38
query34	15.31	2.20	1.78
query35	1.72	1.72	1.69
query36	0.48	0.28	0.29
query37	0.12	0.04	0.04
query38	0.06	0.04	0.04
query39	0.04	0.03	0.03
query40	0.11	0.07	0.08
query41	0.14	0.02	0.02
query42	0.04	0.02	0.02
query43	0.03	0.02	0.03
Total cold run time: 92.71 s
Total hot run time: 14.61 s

hello-stephen · 2026-08-16T20:41:35Z

BE Regression && UT Coverage Report

Increment line coverage 100% (0/0) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	74.87% (32101/42878)
Line Coverage	59.35% (356490/600617)
Region Coverage	55.58% (296940/534218)
Branch Coverage	56.45% (133716/236896)

fixed

### What problem does this PR solve? Issue Number: None Related PR: #41320 Load error-log writing and periodic status reporting can concurrently access the same `std::ofstream`. The reporting path may close and upload the stream while the writer is appending an error row, causing a data race and a BE SIGSEGV in the stream write path. This PR uses `_load_error_log_lock` to serialize stream creation and writes and to protect the first-error and path state. A separate `_s3_error_log_file_lock` serializes remote upload and URL publication, while synchronous S3 I/O runs without holding the load-error writer lock.

Cherry-picked from #66250 Co-authored-by: Refrain <refrain520cc@gmail.com>

0AyanamiRei marked this pull request as ready for review July 30, 2026 02:47

0AyanamiRei requested a review from gavinchou as a code owner July 30, 2026 02:47

github-actionsBot requested changes Jul 30, 2026
View reviewed changes

Comment threadbe/src/runtime/runtime_state.cpp
Comment threadbe/src/runtime/runtime_state.cpp Outdated
Comment threadregression-test/suites/fault_injection_p0/cloud/test_load_error_log_concurrent_write.groovy Outdated

0AyanamiRei added 2 commits July 31, 2026 16:45

add lock
f2c9cf9

Merge branch 'master' into agent/fix-load-error-log-race
6ddee4c

github-actionsBot reviewed Aug 3, 2026
View reviewed changes

github-actionsBot previously requested changes Aug 13, 2026
View reviewed changes

Comment threadbe/src/runtime/runtime_state.cpp

0AyanamiRei force-pushed the agent/fix-load-error-log-race branch from d52664c to 3beb646Compare August 16, 2026 16:47

sollhui approved these changes Aug 18, 2026
View reviewed changes

sollhui added dev/4.1.x dev/4.0.x dev/3.1.x labels Aug 18, 2026

sollhui merged commit 4a49d34 into apache:masterAug 18, 2026
34 of 35 checks passed

github-actionsBot mentioned this pull request Aug 18, 2026
branch-4.1: [fix](be) Synchronize load error log access #66250 #66887
Merged

yiguolei pushed a commit that referenced this pull request Aug 20, 2026
branch-4.1: [fix](be) Synchronize load error log access#66250(#66887)
f99471b
Cherry-picked from #66250 Co-authored-by: Refrain <refrain520cc@gmail.com>

yiguolei added dev/4.1.4-merged and removed dev/4.1.x labels Aug 20, 2026

yiguolei mentioned this pull request Sep 1, 2026
4.1.4 Release Note #67355
Open

Uh oh!

Conversation

0AyanamiRei commented Jul 30, 2026• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What problem does this PR solve?

Release note

Check List (For Author)

Uh oh!

hello-stephen commented Jul 30, 2026

Uh oh!

sollhui commented Jul 30, 2026

Uh oh!

github-actionsBot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

0AyanamiRei commented Aug 3, 2026

Uh oh!

hello-stephen commented Aug 3, 2026

BE UT Coverage Report

Uh oh!

hello-stephen commented Aug 3, 2026

BE Regression && UT Coverage Report

Uh oh!

sollhui commented Aug 3, 2026

Uh oh!

hello-stephen commented Aug 3, 2026

Uh oh!

hello-stephen commented Aug 3, 2026

Uh oh!

hello-stephen commented Aug 3, 2026

Uh oh!

github-actionsBot left a comment

Choose a reason for hiding this comment

Uh oh!

0AyanamiRei commented Aug 4, 2026

Uh oh!

hello-stephen commented Aug 4, 2026

Uh oh!

hello-stephen commented Aug 4, 2026

Uh oh!

hello-stephen commented Aug 4, 2026

Uh oh!

hello-stephen commented Aug 4, 2026

BE UT Coverage Report

Uh oh!

hello-stephen commented Aug 4, 2026

BE Regression && UT Coverage Report

Uh oh!

sollhui commented Aug 13, 2026

Uh oh!

github-actionsBot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

0AyanamiRei commented Aug 16, 2026

Uh oh!

hello-stephen commented Aug 16, 2026

BE UT Coverage Report

Uh oh!

hello-stephen commented Aug 16, 2026

Uh oh!

hello-stephen commented Aug 16, 2026

Uh oh!

hello-stephen commented Aug 16, 2026

Uh oh!

hello-stephen commented Aug 16, 2026

BE Regression && UT Coverage Report

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

0AyanamiRei commented Jul 30, 2026•
edited
Loading