[fix](be) Init thread context on AsyncIO worker threads by liaoxin01 · Pull Request #64846 · apache/doris

liaoxin01 · 2026-06-25T08:34:14Z

Proposed changes

Problem

FILESYSTEM_M dispatches IO ops from a bthread to an AsyncIO worker pthread that is not bound to any task and therefore has no ThreadContext. When the dispatched fn() reads a local file, it goes through LocalFileReader::read_at_impl -> LIMIT_LOCAL_SCAN_IO -> thread_context(), which raises Status::FatalError (aborting the BE in debug/ASAN builds) when no ThreadContext exists.

Observed as a Cloud P0 coredump on the load error-log S3 upload path:

RuntimeState::get_error_log_file_path()
-> _s3_error_fs->upload() (called from a bthread)
-> AsyncIO::run_task() on an AsyncIO worker pthread (no attached task)
-> S3FileSystem::upload_impl()
-> LocalFileReader::read_at_impl()
-> LIMIT_LOCAL_SCAN_IO -> thread_context() -> FatalError -> SIGABRT

Fix

Initialize an (unattached) ThreadContext at the AsyncIO worker boundary via SCOPED_INIT_THREAD_CONTEXT(), the same pattern used by StorageEngine background threads. This gives every thread_context() consumer running on an AsyncIO worker (memory tracking, LIMIT_*_SCAN_IO, ...) a valid context, fixing the crash at its root rather than making individual consumers tolerate a missing context.

hello-stephen · 2026-06-25T08:34:23Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

What problem was fixed (it's best to include specific error reporting information). How it was fixed.
Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
What features were added. Why was this function added?
Which code was refactored and why was this part of the code refactored?
Which functions were optimized and what is the difference before and after the optimization?

Copilot

Pull request overview

This PR aims to prevent BE aborts when RemoteFileSystem::upload() is invoked from a bthread and the call is dispatched via FILESYSTEM_M onto an AsyncIO worker pthread without a ThreadContext, causing LocalFileReader::read_at_impl() (via LIMIT_LOCAL_SCAN_IO) to throw Status::FatalError.

Changes:

Add SCOPED_ATTACH_TASK(ExecEnv::GetInstance()->s3_file_buffer_tracker()) at the start of S3FileSystem::upload_impl() to ensure a task/context exists for local reads.

Review Checkpoints (Part 1.3)

Goal & correctness: Not fully achieved as-is: the new unconditional SCOPED_ATTACH_TASK can overwrite/detach an already-attached task when FILESYSTEM_M runs inline on pthreads, potentially breaking the caller’s thread context and memory tracking.
Change scope: Small and focused to upload_impl(), but the current approach needs to be adjusted to be safe in both inline (pthread) and AsyncIO (pthread-from-bthread) execution modes.
Concurrency/threading: Yes—this code explicitly crosses bthread→pthread via AsyncIO::run_task and must preserve/avoid clobbering existing task attachment on inline pthread paths.
Parallel paths:batch_upload_impl() already attaches; single-file upload_impl() needed equivalent protection, but must be done conditionally to avoid double-attach.
Test coverage: There are existing BE unit tests for S3FileSystem (including upload), but this change should ideally be covered by a test that calls upload() while a task is already attached, to ensure the thread context remains intact.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

liaoxin01 · 2026-06-25T08:46:09Z

/review

github-actions

Summary opinion: I am leaving this as a review comment, not an approval. The only substantiated correctness issue I found is already covered by the existing inline thread on be/src/io/fs/s3_file_system.cpp:329: the new unconditional SCOPED_ATTACH_TASK can run on an already-attached pthread because FILESYSTEM_M executes inline when bthread_self() == 0, and AttachTask is not a safe nested/switching context. I did not add a duplicate inline comment.

Critical checkpoint conclusions:

Goal/correctness: The patch addresses the bthread-to-AsyncIO missing ThreadContext crash path for LocalFileReader::read_at_impl, but it is not fully correct until the existing inline issue is fixed for inline pthread callers.
Scope: The change is small and focused to S3FileSystem::upload_impl.
Concurrency/lifecycle: This path is thread-context-sensitive. AttachTask detaches at scope exit and ThreadContext::attach_task expects no existing resource context; snapshot upload provides a concrete already-attached pthread caller.
Config/compatibility/storage format: No new config, FE-BE protocol, function symbol, or storage-format compatibility issue was introduced.
Parallel paths: batch_upload_impl already attaches inside its S3 thread-pool callback. HDFS/Broker upload paths were checked and did not produce a distinct changed-line finding for this PR.
Tests: No tests were added or changed. Existing upload tests exercise upload, but not the attached-context or bthread-to-AsyncIO mode. git diff --check ba778aa4be4218eb4346d15bcd08d53a6db5affa..a520262d5b1b8a64a2bee675951a749f18492670 -- was clean. I did not run BE tests in this checkout.
Observability/performance/transactions: No additional observability, performance, transaction, or persistence issue was found beyond the task-context bug already raised.

Subagent conclusions:

optimizer-rewrite: no optimizer/rewrite, join, aggregate, or semantic-equivalence finding; noted the task-context issue is duplicate of the existing thread.
tests-session-config: no new test/session/config/style finding; noted the same issue is already covered and git diff --check is clean.
Final convergence round 1: both live subagents replied NO_NEW_VALUABLE_FINDINGS for the same current ledger and proposed final comment set.

User focus: no additional user-provided review focus was supplied.

liaoxin01 · 2026-06-25T15:58:39Z

/review

github-actions

Summary opinion: request changes. The fix is directionally correct for the missing ThreadContext crash path, but the PR does not add focused BE coverage for the exact raw-thread behavior it now relies on.

Critical checkpoint conclusions:

Goal/correctness: The code changes LIMIT_LOCAL_SCAN_IO and LIMIT_REMOTE_SCAN_IO to use thread_context_or_null() so raw pthreads skip workload-group throttling instead of throwing FatalError. That matches the reported FILESYSTEM_M -> AsyncIO::run_task upload path, but the new behavior is untested.
Scope: The code change is small and focused to be/src/runtime/thread_context.h.
Concurrency/lifecycle: This path is thread-context-sensitive. FILESYSTEM_M runs inline on pthread callers and dispatches bthread callers to raw AsyncIO pthreads; the current helper avoids the unsafe unconditional SCOPED_ATTACH_TASK approach already raised in the existing thread.
Config/compatibility/storage format: No new config, FE-BE protocol, function symbol, or storage-format compatibility issue was introduced.
Parallel paths: Both local and remote scan throttle macros were updated. I checked local, S3, HDFS, buffered, and peer-cache reader call sites and found no additional distinct changed-line issue.
Tests: No test was added. Existing thread_context_test runs under the BE test runner's main-thread SCOPED_INIT_THREAD_CONTEXT(), so it does not cover a raw thread with no context. git diff --check ba778aa4be4218eb4346d15bcd08d53a6db5affa 274cc5d3cd62345d20cd9d9f216505636d6e626f is clean.
Observability/performance/transactions: No additional issue found; the change does not add persistence or transaction behavior.

Subagent conclusions:

optimizer-rewrite: no optimizer/rewrite, semantic-equivalence, join, or aggregate candidate; no inline comment from this pass.
tests-session-config: proposed TSC-1 for missing focused raw-thread coverage; accepted as inline comment C-1.
Final convergence round 1: both live subagents replied NO_NEW_VALUABLE_FINDINGS for the same current ledger and proposed final comment set after C-1 was accepted.

User focus: no additional user-provided review focus was supplied.

liaoxin01 · 2026-06-26T03:10:52Z

run buildall

FILESYSTEM_M dispatches IO ops (e.g. S3 single-file upload) from a bthread to an AsyncIO worker pthread that is not bound to any task and therefore has no ThreadContext. When the dispatched fn() reads a local file it goes through LocalFileReader::read_at_impl -> LIMIT_LOCAL_SCAN_IO -> thread_context(), which raises Status::FatalError (aborting the BE in debug/ASAN builds) when no ThreadContext exists. Observed as a Cloud P0 coredump on the load error-log S3 upload path: RuntimeState::get_error_log_file_path() -> _s3_error_fs->upload() (from a bthread) -> AsyncIO::run_task() on an AsyncIO worker pthread (no attached task) -> S3FileSystem::upload_impl() -> LocalFileReader::read_at_impl() -> LIMIT_LOCAL_SCAN_IO -> thread_context() -> FatalError -> SIGABRT Initialize an (unattached) ThreadContext at the AsyncIO worker boundary via SCOPED_INIT_THREAD_CONTEXT(), the same pattern used by StorageEngine background threads, so any thread_context() consumer running on the worker is safe.

liaoxin01 · 2026-06-26T13:33:30Z

/review

liaoxin01 · 2026-06-26T13:33:42Z

run buildall

github-actions

Summary opinion: no new inline comments for the current head. The current one-file fix moves the ThreadContext initialization to the AsyncIO worker boundary, which matches the reported bthread -> AsyncIO pthread crash path better than the earlier S3-local attach workaround. I am not duplicating the already-known coverage concern from discussion_r3475864070, and the older S3 attach-task concern from discussion_r3473013312 no longer applies to the current changed file.

Critical checkpoint conclusions:

Goal/correctness: The current patch adds SCOPED_INIT_THREAD_CONTEXT() inside AsyncIO::run_task, so FILESYSTEM_M-dispatched work and LocalFileReader close work on AsyncIO pthreads have a ThreadContext before code such as LIMIT_LOCAL_SCAN_IO calls thread_context(). I found no new correctness issue in the current changed file.
Scope: The GitHub PR file list contains only be/src/util/async_io.h, and the change is small and focused.
Concurrency/lifecycle: This is a thread-context boundary. The new scope creates an unattached context only for the worker callback and tears it down after fn() returns; nested SCOPED_ATTACH_TASK paths restore through the existing ThreadMemTrackerMgr snapshot logic. No new lock-order or deadlock issue was found.
Config/compatibility/storage format: No new config, FE-BE protocol field, function symbol, or storage-format compatibility issue is introduced.
Parallel paths: FILESYSTEM_M dispatch and LocalFileReader::close both use AsyncIO::run_task, so the worker-boundary fix covers the relevant AsyncIO callers. The previous single S3 upload attach change is absent from this head; batch upload keeps its existing explicit task attach in its own thread-pool callback.
Tests: No tests were added. The missing focused raw-worker/attached-context coverage concern substantially overlaps the existing inline thread discussion_r3475864070, so I did not submit a duplicate. I did not run BE tests because thirdparty/installed is absent in this checkout. git diff --check for be/src/util/async_io.h on the PR base/head range was clean.
Observability/performance/transactions/persistence: No additional observability, performance, transaction, or persistence issue was found for this change.

Subagent conclusions:

optimizer-rewrite: no optimizer/rewrite, semantic-equivalence, join, or aggregate candidate; duplicate context only for the existing S3 attach-task and test-coverage threads.
tests-session-config: no new tests/session/config/style candidate; the remaining coverage concern is duplicate of discussion_r3475864070.
Final convergence round 1: both live subagents replied NO_NEW_VALUABLE_FINDINGS for the same current ledger and empty proposed inline comment set after the main merge update.

User focus: no additional user-provided review focus was supplied.

fixed

hello-stephen · 2026-06-26T14:03:35Z

TPC-H: Total hot run time: 28632 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit f15b39dfb1559867227c0594c89de85f0cb45c6b, data reload: false
------ Round 1 ----------------------------------
============================================
q1	17707	4044	4005	4005
q2	2011	297	185	185
q3	10322	1380	836	836
q4	4699	462	338	338
q5	7630	871	552	552
q6	201	176	140	140
q7	766	829	619	619
q8	9885	1457	1457	1457
q9	5971	4528	4495	4495
q10	6810	1806	1509	1509
q11	440	272	243	243
q12	664	425	304	304
q13	18147	3377	2719	2719
q14	274	257	238	238
q15	q16	783	779	703	703
q17	1195	945	783	783
q18	6804	5752	5504	5504
q19	1415	1350	1083	1083
q20	487	391	260	260
q21	5898	2669	2361	2361
q22	434	358	298	298
Total cold run time: 102543 ms
Total hot run time: 28632 ms
----- Round 2, with runtime_filter_mode=off -----
============================================
q1	4375	4287	4290	4287
q2	329	347	223	223
q3	4624	4961	4394	4394
q4	2110	2167	1389	1389
q5	4457	4332	4296	4296
q6	235	177	131	131
q7	1734	1613	1993	1613
q8	2514	2202	2189	2189
q9	8117	8111	8109	8109
q10	4793	4710	4318	4318
q11	611	424	374	374
q12	751	887	577	577
q13	3342	3519	3006	3006
q14	303	308	276	276
q15	q16	732	742	639	639
q17	1380	1350	1334	1334
q18	8075	7295	6890	6890
q19	1136	1148	1123	1123
q20	2202	2240	1961	1961
q21	5308	4591	4520	4520
q22	521	474	407	407
Total cold run time: 57649 ms
Total hot run time: 52056 ms

hello-stephen · 2026-06-26T14:14:32Z

TPC-DS: Total hot run time: 172059 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit f15b39dfb1559867227c0594c89de85f0cb45c6b, data reload: false
query5	4305	633	474	474
query6	464	188	172	172
query7	4852	572	315	315
query8	320	180	163	163
query9	8765	4152	4123	4123
query10	480	316	264	264
query11	5960	2328	2126	2126
query12	154	104	106	104
query13	1314	608	451	451
query14	6483	5369	4984	4984
query14_1	4317	4344	4317	4317
query15	220	199	181	181
query16	1012	473	402	402
query17	1104	713	558	558
query18	2690	462	336	336
query19	196	187	140	140
query20	115	108	106	106
query21	211	149	119	119
query22	13635	13630	13417	13417
query23	17327	16503	16101	16101
query23_1	16290	16356	16438	16356
query24	8148	1773	1303	1303
query24_1	1341	1372	1330	1330
query25	530	454	378	378
query26	1334	321	177	177
query27	3477	561	345	345
query28	5000	2105	2064	2064
query29	1075	626	489	489
query30	309	234	205	205
query31	1143	1085	960	960
query32	111	66	59	59
query33	531	326	256	256
query34	1178	1190	681	681
query35	785	802	712	712
query36	1466	1466	1276	1276
query37	161	114	98	98
query38	2083	1742	1702	1702
query39	965	953	922	922
query39_1	898	909	934	909
query40	225	129	111	111
query41	75	69	69	69
query42	93	91	94	91
query43	330	328	291	291
query44	1497	790	795	790
query45	207	199	185	185
query46	1081	1222	767	767
query47	2351	2317	2244	2244
query48	404	427	304	304
query49	586	455	312	312
query50	1008	371	273	273
query51	4459	4396	4348	4348
query52	84	86	73	73
query53	258	270	199	199
query54	295	229	226	226
query55	77	72	66	66
query56	250	247	227	227
query57	1412	1419	1310	1310
query58	243	222	207	207
query59	1594	1632	1457	1457
query60	286	258	241	241
query61	167	164	173	164
query62	690	656	597	597
query63	241	194	197	194
query64	2527	801	654	654
query65	4898	4780	4826	4780
query66	1751	453	333	333
query67	28865	28897	28722	28722
query68	3049	1599	901	901
query69	422	315	265	265
query70	1036	957	960	957
query71	291	255	215	215
query72	2922	2593	2360	2360
query73	833	764	438	438
query74	5166	4915	4787	4787
query75	2547	2549	2194	2194
query76	2351	1233	789	789
query77	373	400	286	286
query78	12333	12508	11738	11738
query79	1369	1224	783	783
query80	679	479	381	381
query81	463	278	239	239
query82	579	158	125	125
query83	314	286	254	254
query84	267	147	113	113
query85	906	539	418	418
query86	376	306	281	281
query87	1865	1821	1767	1767
query88	3768	2772	2758	2758
query89	437	383	346	346
query90	1889	192	189	189
query91	175	164	138	138
query92	64	59	62	59
query93	1448	1567	909	909
query94	559	343	314	314
query95	694	382	434	382
query96	1106	816	340	340
query97	2691	2683	2585	2585
query98	215	231	203	203
query99	1175	1163	1027	1027
Total cold run time: 259452 ms
Total hot run time: 172059 ms

hello-stephen · 2026-06-26T14:19:28Z

ClickBench: Total hot run time: 25.13 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit f15b39dfb1559867227c0594c89de85f0cb45c6b, data reload: false
query1	0.01	0.01	0.00
query2	0.10	0.05	0.05
query3	0.26	0.14	0.13
query4	1.61	0.15	0.14
query5	0.24	0.25	0.22
query6	1.25	1.10	1.03
query7	0.04	0.01	0.00
query8	0.06	0.04	0.04
query9	0.38	0.31	0.33
query10	0.59	0.55	0.54
query11	0.20	0.15	0.14
query12	0.18	0.15	0.15
query13	0.47	0.47	0.48
query14	1.01	1.01	1.02
query15	0.62	0.58	0.59
query16	0.32	0.33	0.32
query17	1.05	1.14	1.14
query18	0.22	0.20	0.19
query19	2.08	1.95	1.91
query20	0.02	0.01	0.02
query21	15.43	0.24	0.13
query22	4.74	0.06	0.05
query23	16.12	0.33	0.13
query24	2.98	0.42	0.32
query25	0.12	0.04	0.04
query26	0.73	0.22	0.15
query27	0.04	0.04	0.04
query28	3.58	0.92	0.51
query29	12.52	4.34	3.46
query30	0.27	0.16	0.16
query31	2.77	0.60	0.31
query32	3.22	0.60	0.49
query33	3.21	3.18	3.26
query34	15.48	4.23	3.51
query35	3.55	3.50	3.56
query36	0.56	0.44	0.41
query37	0.09	0.07	0.06
query38	0.05	0.04	0.03
query39	0.03	0.04	0.03
query40	0.18	0.15	0.15
query41	0.08	0.04	0.03
query42	0.04	0.03	0.03
query43	0.05	0.04	0.03
Total cold run time: 96.55 s
Total hot run time: 25.13 s

liaoxin01 · 2026-06-29T06:58:31Z

run buildall

hello-stephen · 2026-06-29T07:23:15Z

TPC-H: Total hot run time: 29306 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit f15b39dfb1559867227c0594c89de85f0cb45c6b, data reload: false
------ Round 1 ----------------------------------
============================================
q1	17621	4082	4034	4034
q2	2009	313	194	194
q3	10305	1447	812	812
q4	4680	478	336	336
q5	7524	865	579	579
q6	184	170	139	139
q7	774	841	632	632
q8	9350	1570	1643	1570
q9	5619	4545	4522	4522
q10	6747	1788	1500	1500
q11	439	272	240	240
q12	633	417	287	287
q13	18101	3349	2797	2797
q14	271	253	246	246
q15	q16	788	777	717	717
q17	967	1057	1090	1057
q18	6929	5838	5567	5567
q19	1295	1312	1121	1121
q20	520	414	260	260
q21	5996	2675	2397	2397
q22	426	360	299	299
Total cold run time: 101178 ms
Total hot run time: 29306 ms
----- Round 2, with runtime_filter_mode=off -----
============================================
q1	4309	4248	4242	4242
q2	327	349	218	218
q3	4594	4983	4372	4372
q4	2074	2167	1370	1370
q5	4438	4321	4317	4317
q6	230	175	128	128
q7	1752	1652	1919	1652
q8	2623	2199	2196	2196
q9	8230	8451	8138	8138
q10	4786	4720	4290	4290
q11	581	406	387	387
q12	743	758	542	542
q13	3220	3523	2962	2962
q14	306	300	265	265
q15	q16	712	740	623	623
q17	1364	1317	1321	1317
q18	7950	7281	7071	7071
q19	1144	1129	1144	1129
q20	2190	2219	1978	1978
q21	5292	4590	4480	4480
q22	504	456	401	401
Total cold run time: 57369 ms
Total hot run time: 52078 ms

hello-stephen · 2026-06-29T07:34:10Z

TPC-DS: Total hot run time: 171897 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit f15b39dfb1559867227c0594c89de85f0cb45c6b, data reload: false
query5	4314	629	476	476
query6	429	195	168	168
query7	4895	555	311	311
query8	333	182	170	170
query9	8801	4044	4017	4017
query10	439	315	252	252
query11	5919	2345	2181	2181
query12	152	103	100	100
query13	1256	601	439	439
query14	6273	5293	4970	4970
query14_1	4302	4270	4290	4270
query15	237	199	179	179
query16	1013	461	438	438
query17	1116	718	574	574
query18	2695	470	334	334
query19	200	183	139	139
query20	108	106	103	103
query21	219	137	115	115
query22	13684	13596	13395	13395
query23	17526	16568	16157	16157
query23_1	16210	16225	16258	16225
query24	7431	1818	1287	1287
query24_1	1333	1308	1297	1297
query25	534	436	375	375
query26	1286	295	170	170
query27	2667	570	351	351
query28	4385	2004	2011	2004
query29	1071	599	475	475
query30	309	228	195	195
query31	1111	1077	962	962
query32	114	66	70	66
query33	557	338	267	267
query34	1167	1105	622	622
query35	779	808	674	674
query36	1393	1381	1246	1246
query37	154	109	94	94
query38	1905	1713	1686	1686
query39	936	913	906	906
query39_1	875	870	905	870
query40	228	132	105	105
query41	71	68	71	68
query42	90	91	89	89
query43	319	321	283	283
query44	1472	803	784	784
query45	207	194	181	181
query46	1066	1257	770	770
query47	2381	2363	2276	2276
query48	415	421	299	299
query49	600	428	332	332
query50	991	344	260	260
query51	4393	4396	4388	4388
query52	83	83	71	71
query53	255	271	192	192
query54	290	248	236	236
query55	76	70	67	67
query56	247	254	240	240
query57	1427	1392	1343	1343
query58	265	233	222	222
query59	1594	1655	1479	1479
query60	300	263	231	231
query61	181	169	174	169
query62	698	654	596	596
query63	230	196	204	196
query64	2544	867	581	581
query65	4881	4770	4783	4770
query66	1769	463	354	354
query67	29010	28761	28797	28761
query68	3190	1614	835	835
query69	408	307	246	246
query70	1076	955	954	954
query71	287	234	219	219
query72	2849	2604	2309	2309
query73	812	761	450	450
query74	5133	4937	4794	4794
query75	2554	2540	2171	2171
query76	2358	1185	772	772
query77	351	372	284	284
query78	12478	12498	11917	11917
query79	1432	1211	759	759
query80	945	461	382	382
query81	499	273	250	250
query82	574	161	124	124
query83	323	279	254	254
query84	255	142	116	116
query85	892	510	417	417
query86	411	310	272	272
query87	1843	1836	1775	1775
query88	3670	2777	2752	2752
query89	439	377	332	332
query90	1851	191	185	185
query91	179	163	134	134
query92	67	60	54	54
query93	1493	1457	992	992
query94	611	351	336	336
query95	664	453	344	344
query96	1020	796	353	353
query97	2722	2708	2547	2547
query98	212	206	200	200
query99	1193	1145	1034	1034
Total cold run time: 257502 ms
Total hot run time: 171897 ms

hello-stephen · 2026-06-29T07:39:06Z

ClickBench: Total hot run time: 25.45 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit f15b39dfb1559867227c0594c89de85f0cb45c6b, data reload: false
query1	0.01	0.01	0.00
query2	0.09	0.05	0.05
query3	0.25	0.14	0.16
query4	1.61	0.13	0.13
query5	0.25	0.23	0.22
query6	1.19	1.12	1.07
query7	0.03	0.01	0.01
query8	0.06	0.04	0.04
query9	0.41	0.32	0.33
query10	0.57	0.58	0.64
query11	0.19	0.15	0.14
query12	0.18	0.14	0.14
query13	0.49	0.49	0.48
query14	1.01	1.02	0.99
query15	0.62	0.62	0.60
query16	0.32	0.34	0.32
query17	1.14	1.10	1.10
query18	0.22	0.21	0.21
query19	2.09	2.02	1.98
query20	0.02	0.01	0.01
query21	15.45	0.23	0.14
query22	4.87	0.05	0.05
query23	16.15	0.31	0.13
query24	2.97	0.44	0.32
query25	0.12	0.04	0.04
query26	0.73	0.21	0.16
query27	0.04	0.04	0.03
query28	3.53	0.94	0.54
query29	12.52	4.31	3.48
query30	0.27	0.15	0.16
query31	2.78	0.60	0.32
query32	3.23	0.61	0.49
query33	3.26	3.28	3.25
query34	15.60	4.19	3.52
query35	3.54	3.52	3.56
query36	0.55	0.44	0.41
query37	0.08	0.06	0.07
query38	0.05	0.04	0.04
query39	0.04	0.03	0.03
query40	0.18	0.16	0.15
query41	0.08	0.03	0.03
query42	0.03	0.03	0.03
query43	0.04	0.03	0.04
Total cold run time: 96.86 s
Total hot run time: 25.45 s

hello-stephen · 2026-06-29T09:33:25Z

BE Regression && UT Coverage Report

Increment line coverage 100.00% (1/1) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	74.19% (28495/38406)
Line Coverage	58.05% (310142/534248)
Region Coverage	54.80% (259409/473339)
Branch Coverage	56.17% (112720/200669)

hello-stephen · 2026-06-29T09:48:54Z

BE Regression && UT Coverage Report

Increment line coverage 100.00% (1/1) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	74.19% (28495/38406)
Line Coverage	58.05% (310156/534248)
Region Coverage	54.81% (259428/473339)
Branch Coverage	56.18% (112728/200669)

github-actions · 2026-06-29T13:08:43Z

PR approved by at least one committer and no changes requested.

github-actions · 2026-06-29T13:08:45Z

PR approved by anyone and no changes requested.

## Proposed changes ### Problem `FILESYSTEM_M` dispatches IO ops from a bthread to an `AsyncIO` worker pthread that is **not bound to any task** and therefore has no `ThreadContext`. When the dispatched `fn()` reads a local file, it goes through `LocalFileReader::read_at_impl` -> `LIMIT_LOCAL_SCAN_IO` -> `thread_context()`, which raises `Status::FatalError` (aborting the BE in debug/ASAN builds) when no `ThreadContext` exists. Observed as a Cloud P0 coredump on the load error-log S3 upload path: ``` RuntimeState::get_error_log_file_path() -> _s3_error_fs->upload() (called from a bthread) -> AsyncIO::run_task() on an AsyncIO worker pthread (no attached task) -> S3FileSystem::upload_impl() -> LocalFileReader::read_at_impl() -> LIMIT_LOCAL_SCAN_IO -> thread_context() -> FatalError -> SIGABRT ``` ### Fix Initialize an (unattached) `ThreadContext` at the `AsyncIO` worker boundary via `SCOPED_INIT_THREAD_CONTEXT()`, the same pattern used by StorageEngine background threads. This gives every `thread_context()` consumer running on an AsyncIO worker (memory tracking, `LIMIT_*_SCAN_IO`, ...) a valid context, fixing the crash at its root rather than making individual consumers tolerate a missing context.

…64846 (#64970) Cherry-picked from #64846 Co-authored-by: Xin Liao <liaoxin@selectdb.com>

…65271) Pick #64846

liaoxin01 requested review from Gabriel39, gavinchou and morningman as code owners June 25, 2026 08:34

CopilotAI review requested due to automatic review settings June 25, 2026 08:34

Copilotstarted reviewing on behalf of liaoxin01 June 25, 2026 08:34 View session

CopilotAI reviewed Jun 25, 2026
View reviewed changes

Comment threadbe/src/io/fs/s3_file_system.cpp Outdated

liaoxin01 marked this pull request as draft June 25, 2026 08:52

github-actionsBot reviewed Jun 25, 2026
View reviewed changes

liaoxin01 force-pushed the fix-s3-upload-impl-thread-context branch from a520262 to 274cc5dCompare June 25, 2026 08:59

liaoxin01 changed the title ~~[fix](s3) Attach task in S3FileSystem::upload_impl to avoid thread context crash~~[fix](be) Make local/remote scan IO throttle tolerate missing thread contextJun 25, 2026

github-actionsBot previously requested changes Jun 25, 2026
View reviewed changes

Comment threadbe/src/runtime/thread_context.h Outdated

liaoxin01 force-pushed the fix-s3-upload-impl-thread-context branch from 274cc5d to f15b39dCompare June 26, 2026 13:30

liaoxin01 changed the title ~~[fix](be) Make local/remote scan IO throttle tolerate missing thread context~~[fix](be) Init thread context on AsyncIO worker threadsJun 26, 2026

liaoxin01 marked this pull request as ready for review June 26, 2026 13:33

github-actionsBot reviewed Jun 26, 2026
View reviewed changes

liaoxin01 added dev/3.1.x dev/4.0.x dev/4.1.x labels Jun 26, 2026

gavinchou approved these changes Jun 29, 2026
View reviewed changes

github-actionsBot added the approved Indicates a PR has been approved by one committer. label Jun 29, 2026

github-actionsBot added the reviewed label Jun 29, 2026

liaoxin01 merged commit 5981cd7 into apache:masterJun 30, 2026
38 of 39 checks passed

liaoxin01 deleted the fix-s3-upload-impl-thread-context branch June 30, 2026 02:27

github-actionsBot added the dev/4.0.x-conflict label Jun 30, 2026

github-actionsBot mentioned this pull request Jun 30, 2026
branch-4.1: [fix](be) Init thread context on AsyncIO worker threads #64846 #64970
Merged

yiguolei pushed a commit that referenced this pull request Jun 30, 2026
branch-4.1: [fix](be) Init thread context on AsyncIO worker threads#…
c76d497
…64846 (#64970) Cherry-picked from #64846 Co-authored-by: Xin Liao <liaoxin@selectdb.com>

yiguolei added dev/4.1.3-merged and removed dev/4.1.x labels Jun 30, 2026

liaoxin01 mentioned this pull request Jul 6, 2026
[branch-4.0][fix](be) Init thread context on AsyncIO worker threads #65271
Merged

yiguolei mentioned this pull request Jul 7, 2026
4.1.3 Release Note #65306
Closed

morningman pushed a commit that referenced this pull request Jul 7, 2026
[branch-4.0][fix](be) Init thread context on AsyncIO worker threads (#…
8251b44
…65271) Pick #64846

morningman added dev/4.0.7-merged and removed dev/4.0.x dev/4.0.x-conflict labels Jul 7, 2026

morningman mentioned this pull request Jul 9, 2026
4.0.7 Release Notes #65399
Closed

liaoxin01 removed the dev/3.1.x label Sep 4, 2026

Uh oh!

Conversation

liaoxin01 commented Jun 25, 2026• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Proposed changes

Problem

Fix

Uh oh!

hello-stephen commented Jun 25, 2026

Uh oh!

CopilotAI left a comment

Choose a reason for hiding this comment

Pull request overview

Review Checkpoints (Part 1.3)

Uh oh!

Uh oh!

liaoxin01 commented Jun 25, 2026

Uh oh!

github-actionsBot left a comment

Choose a reason for hiding this comment

Uh oh!

liaoxin01 commented Jun 25, 2026

Uh oh!

github-actionsBot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

liaoxin01 commented Jun 26, 2026

Uh oh!

liaoxin01 commented Jun 26, 2026

Uh oh!

liaoxin01 commented Jun 26, 2026

Uh oh!

github-actionsBot left a comment

Choose a reason for hiding this comment

Uh oh!

hello-stephen commented Jun 26, 2026

Uh oh!

hello-stephen commented Jun 26, 2026

Uh oh!

hello-stephen commented Jun 26, 2026

Uh oh!

liaoxin01 commented Jun 29, 2026

Uh oh!

hello-stephen commented Jun 29, 2026

Uh oh!

hello-stephen commented Jun 29, 2026

Uh oh!

hello-stephen commented Jun 29, 2026

Uh oh!

hello-stephen commented Jun 29, 2026

BE Regression && UT Coverage Report

Uh oh!

hello-stephen commented Jun 29, 2026

BE Regression && UT Coverage Report

Uh oh!

github-actionsBot commented Jun 29, 2026

Uh oh!

github-actionsBot commented Jun 29, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

liaoxin01 commented Jun 25, 2026•
edited
Loading