[fix](temp-table) not clean temp table temporary until fix mem leak by morrySnow · Pull Request #59535 · apache/doris

morrySnow · 2026-01-04T08:53:30Z

What problem does this PR solve?

Related PR: #40680

Problem Summary:

This pull request temporarily disables session tracking and automatic cleanup for temporary tables due to a memory leak issue involving Env#sessionReportTimeMap and Env#aliveSessionSet. The affected logic is commented out with TODO notes, and related imports are cleaned up.

Session management changes:

Disabled the code that adds sessions to aliveSessionSet in Env.registerSessionInfo, preventing new session tracking until the memory leak is fixed.
Disabled the code that updates session report times in Env.refreshSession, pausing session activity tracking.

Temporary table cleanup changes:

Commented out the logic in TemporaryTableMgr.runAfterCatalogReady that deletes temporary tables when their creating session is gone, halting automatic cleanup.

Release note

None

Check List (For Author)

Test
- Regression test
- Unit Test
- Manual test (add detailed scripts or steps below)
- No need to test or manual test. Explain why:
  - This is a refactor/code format and no logic has been changed.
  - Previous test can cover this change.
  - No code files have been changed.
  - Other reason
Behavior changed:
- No.
- Yes.
Does this need documentation?
- No.
- Yes.

Check List (For Reviewer who merge this PR)

Confirm the release note
Confirm test cases
Confirm document
Add branch pick label

hello-stephen · 2026-01-04T08:53:38Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

What problem was fixed (it's best to include specific error reporting information). How it was fixed.
Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
What features were added. Why was this function added?
Which code was refactored and why was this part of the code refactored?
Which functions were optimized and what is the difference before and after the optimization?

morrySnow · 2026-01-04T08:53:49Z

run buildall

doris-robot · 2026-01-04T09:20:24Z

TPC-H: Total hot run time: 31584 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit a398de38984417a5be8cd4ef235371f4935fd599, data reload: false
------ Round 1 ----------------------------------
q1	17603	4288	4056	4056
q2	2123	357	243	243
q3	10071	1282	740	740
q4	10227	910	336	336
q5	7523	2092	1864	1864
q6	187	172	149	149
q7	931	837	677	677
q8	9283	1378	1174	1174
q9	4925	4623	4565	4565
q10	6740	1786	1409	1409
q11	527	304	280	280
q12	702	778	597	597
q13	17766	3803	3100	3100
q14	286	287	278	278
q15	586	520	506	506
q16	685	683	611	611
q17	641	804	509	509
q18	6705	6637	6378	6378
q19	1080	962	605	605
q20	400	361	246	246
q21	3031	2489	2284	2284
q22	1043	999	977	977
Total cold run time: 103065 ms
Total hot run time: 31584 ms
----- Round 2, with runtime_filter_mode=off -----
q1	4126	4098	4075	4075
q2	345	402	314	314
q3	2049	2605	2221	2221
q4	1344	1758	1336	1336
q5	4188	4052	4069	4052
q6	219	176	131	131
q7	1873	1756	1623	1623
q8	2843	2560	2449	2449
q9	7193	7282	7201	7201
q10	2634	2804	2268	2268
q11	541	488	470	470
q12	741	844	688	688
q13	3624	4038	3403	3403
q14	295	320	278	278
q15	574	505	502	502
q16	668	686	641	641
q17	1165	1321	1430	1321
q18	7981	8002	7602	7602
q19	888	860	852	852
q20	2057	2110	1922	1922
q21	4816	4534	4418	4418
q22	1130	1086	1070	1070
Total cold run time: 51294 ms
Total hot run time: 48837 ms

doris-robot · 2026-01-04T09:31:06Z

TPC-DS: Total hot run time: 172568 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit a398de38984417a5be8cd4ef235371f4935fd599, data reload: false
query5	4935	590	431	431
query6	344	244	207	207
query7	4217	454	279	279
query8	354	230	221	221
query9	8770	2646	2623	2623
query10	529	366	323	323
query11	15357	15111	14949	14949
query12	185	112	113	112
query13	1259	491	364	364
query14	6425	2997	2798	2798
query14_1	2803	2647	2674	2647
query15	202	189	174	174
query16	968	475	486	475
query17	1091	677	589	589
query18	2600	449	346	346
query19	269	214	199	199
query20	129	116	113	113
query21	213	138	116	116
query22	3881	4073	3734	3734
query23	16372	15606	15339	15339
query23_1	15436	15637	15389	15389
query24	7601	1548	1168	1168
query24_1	1188	1169	1200	1169
query25	525	443	377	377
query26	1230	264	148	148
query27	2773	451	293	293
query28	4531	2149	2149	2149
query29	782	514	420	420
query30	305	246	214	214
query31	817	631	585	585
query32	78	69	68	68
query33	542	330	261	261
query34	906	907	530	530
query35	748	795	704	704
query36	820	861	832	832
query37	129	93	79	79
query38	2732	2733	2606	2606
query39	791	762	721	721
query39_1	715	723	726	723
query40	219	133	115	115
query41	66	65	61	61
query42	104	105	101	101
query43	478	469	428	428
query44	1329	741	731	731
query45	191	184	187	184
query46	844	968	579	579
query47	1393	1401	1392	1392
query48	310	319	268	268
query49	604	414	318	318
query50	635	277	229	229
query51	3916	3771	3753	3753
query52	100	104	93	93
query53	289	328	280	280
query54	297	258	235	235
query55	76	75	74	74
query56	297	293	285	285
query57	962	1007	934	934
query58	265	253	257	253
query59	2026	2148	1978	1978
query60	312	329	300	300
query61	174	158	164	158
query62	397	342	332	332
query63	297	265	269	265
query64	5039	1297	976	976
query65	3756	3740	3733	3733
query66	1405	427	313	313
query67	14924	15115	14658	14658
query68	2678	1013	746	746
query69	446	361	323	323
query70	1001	907	947	907
query71	308	304	276	276
query72	6175	3440	3426	3426
query73	590	728	309	309
query74	8868	8783	8578	8578
query75	2754	2832	2483	2483
query76	2945	1055	642	642
query77	324	362	285	285
query78	9666	9880	9196	9196
query79	980	920	591	591
query80	632	581	504	504
query81	496	263	223	223
query82	220	142	108	108
query83	264	256	237	237
query84	265	127	106	106
query85	861	513	468	468
query86	349	290	317	290
query87	2868	2872	2833	2833
query88	3169	2234	2232	2232
query89	389	346	342	342
query90	2024	152	156	152
query91	170	159	141	141
query92	91	77	64	64
query93	920	872	539	539
query94	463	332	295	295
query95	555	371	305	305
query96	593	460	210	210
query97	2373	2383	2289	2289
query98	214	196	192	192
query99	628	578	576	576
Total cold run time: 247522 ms
Total hot run time: 172568 ms

doris-robot · 2026-01-04T09:36:10Z

ClickBench: Total hot run time: 26.93 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit a398de38984417a5be8cd4ef235371f4935fd599, data reload: false
query1	0.05	0.05	0.05
query2	0.11	0.05	0.05
query3	0.27	0.09	0.08
query4	1.61	0.12	0.11
query5	0.27	0.27	0.25
query6	1.15	0.67	0.65
query7	0.03	0.02	0.02
query8	0.05	0.03	0.04
query9	0.57	0.50	0.51
query10	0.56	0.55	0.54
query11	0.15	0.10	0.10
query12	0.14	0.10	0.10
query13	0.61	0.59	0.58
query14	0.96	0.94	0.95
query15	0.80	0.77	0.80
query16	0.38	0.42	0.40
query17	1.04	1.00	1.06
query18	0.23	0.21	0.21
query19	1.86	1.86	1.77
query20	0.02	0.01	0.01
query21	15.53	0.24	0.15
query22	6.57	0.06	0.05
query23	16.18	0.29	0.10
query24	0.96	0.68	0.37
query25	0.12	0.07	0.05
query26	0.14	0.14	0.14
query27	0.06	0.07	0.08
query28	4.49	1.05	0.88
query29	12.64	3.97	3.21
query30	0.29	0.16	0.12
query31	2.84	0.63	0.39
query32	3.23	0.58	0.46
query33	3.01	3.04	3.00
query34	16.48	5.13	4.49
query35	4.46	4.50	4.45
query36	0.66	0.50	0.49
query37	0.10	0.07	0.06
query38	0.06	0.04	0.03
query39	0.05	0.03	0.03
query40	0.17	0.14	0.14
query41	0.13	0.04	0.03
query42	0.04	0.03	0.03
query43	0.04	0.03	0.03
Total cold run time: 99.11 s
Total hot run time: 26.93 s

hello-stephen · 2026-01-04T10:14:02Z

FE UT Coverage Report

Increment line coverage 100% (0/0) 🎉
Increment coverage report
Complete coverage report

hello-stephen · 2026-01-04T13:09:19Z

FE Regression Coverage Report

Increment line coverage 100% (0/0) 🎉
Increment coverage report
Complete coverage report

dataroaring

LGTM

github-actions · 2026-01-05T03:16:33Z

PR approved by at least one committer and no changes requested.

github-actions · 2026-01-05T03:16:36Z

PR approved by anyone and no changes requested.

…59535) ### What problem does this PR solve? Related PR: #40680 Problem Summary: This pull request temporarily disables session tracking and automatic cleanup for temporary tables due to a memory leak issue involving `Env#sessionReportTimeMap` and `Env#aliveSessionSet`. The affected logic is commented out with TODO notes, and related imports are cleaned up. Session management changes: * Disabled the code that adds sessions to `aliveSessionSet` in `Env.registerSessionInfo`, preventing new session tracking until the memory leak is fixed. * Disabled the code that updates session report times in `Env.refreshSession`, pausing session activity tracking. Temporary table cleanup changes: * Commented out the logic in `TemporaryTableMgr.runAfterCatalogReady` that deletes temporary tables when their creating session is gone, halting automatic cleanup. ### Release note None ### Check List (For Author) - Test  - [ ] Regression test - [ ] Unit Test - [ ] Manual test (add detailed scripts or steps below) - [x] No need to test or manual test. Explain why: - [ ] This is a refactor/code format and no logic has been changed. - [x] Previous test can cover this change. - [ ] No code files have been changed. - [ ] Other reason  - Behavior changed: - [x] No. - [ ] Yes.  - Does this need documentation? - [x] No. - [ ] Yes.  ### Check List (For Reviewer who merge this PR) - [ ] Confirm the release note - [ ] Confirm test cases - [ ] Confirm document - [ ] Add branch pick label

…x mem leak #59535 (#59551) Cherry-picked from #59535 Co-authored-by: morrySnow <zhangwenxin@selectdb.com>

…porary until fix mem leak apache#59535 (apache#6479) cherry-picks from apache#59535 Related to apache#6477 Co-authored-by: morrySnow <zhangwenxin@selectdb.com>

…x mem leak apache#59535 (apache#59551) Cherry-picked from apache#59535 Co-authored-by: morrySnow <zhangwenxin@selectdb.com>

…rolling upgrade ### What problem does this PR solve? Related PR: apache#59535 Problem Summary: apache#59535 disabled the leaking session-tracking bookkeeping (Env#aliveSessionSet / Env#sessionReportTimeMap) by commenting out registerSessionInfo/refreshSession, so aliveSessionSet always stays empty. During a rolling upgrade, an old-version master FE still queries follower FEs for their alive session ids (getAllAliveSessionIds RPC) and drops temporary tables whose owning session is not reported alive after loss_conn_fe_temp_table_keep_second (default 60s). A new-version follower would report an empty list, so the old master would wrongly delete temp tables of sessions that are actually alive on the new follower. Fix: derive getAllAliveSessionIds() from the live connection registry (ExecuteEnv -> ConnectScheduler -> getConnectionMap, which covers both the MySQL protocol pool and the Arrow Flight SQL pool) instead of the always-empty aliveSessionSet. The registry always reflects real client connections and needs no register/unregister bookkeeping, so alive-session reporting stays accurate for an old-version master while the leak fix of apache#59535 remains intact (registerSessionInfo/refreshSession/TemporaryTableMgr stay commented out). ### Release note None ### Check List (For Author) - Test: Unit Test (EnvTest#testGetAllAliveSessionIds), ./build.sh --fe - Behavior changed: No - Does this need documentation: No

…n disconnect ### What problem does this PR solve? Related PR: apache#59535 Problem Summary: When a client connects to a follower FE and creates a temporary table, the CREATE statement is forwarded to master FE and executed there with a proxy ConnectContext (isProxy=true). Env.registerTempTableAndSession() therefore records the temp table's inner name only in the master-side proxy session, and the original client session's dbToTempTableNamesMap on the follower stays empty. When the follower session later disconnects, times out or is killed, ConnectContext.deleteTempTable() iterates an empty map and drops nothing. apache#59535 disabled the TemporaryTableMgr background cleanup that previously collected such orphan temp tables, so temp tables created via a follower leak permanently (unbounded metadata/tablet/storage leak). Fix: after a statement is successfully forwarded to master FE (StmtExecutor.forwardToMaster), if it is a CREATE TEMPORARY TABLE (nereids CreateTableCommand or legacy CreateTableStmt), register the deterministic inner name (sessionId + sign + tableName, same as master side since the session id is forwarded) into the follower session's dbToTempTableNamesMap; if it is a forwarded DROP TABLE, remove the entry symmetrically so a later disconnect does not re-drop it. Also fix the previously dead non-master branch of ConnectContext.deleteTempTable() to qualify the forwarded DROP with the db name recorded in the map, instead of relying on the session's current database at disconnect time. ### Release note Fix the issue that temporary tables created on a follower FE are not cleaned up after the session disconnects. ### Check List (For Author) - Test: Unit Test (existing org.apache.doris.catalog.EnvTest passed); added docker regression test temp_table_p0/test_temp_table_follower_disconnect (NOT run locally, docker regression environment unavailable) - Behavior changed: Yes - temporary tables created through a follower FE connection are now dropped when the session exits, as originally intended - Does this need documentation: No

…pache#59535) (#93) * [fix](temp-table) not clean temp table temporary until fix mem leak (apache#59535) ### What problem does this PR solve? Related PR: apache#40680 Problem Summary: This pull request temporarily disables session tracking and automatic cleanup for temporary tables due to a memory leak issue involving `Env#sessionReportTimeMap` and `Env#aliveSessionSet`. The affected logic is commented out with TODO notes, and related imports are cleaned up. Session management changes: * Disabled the code that adds sessions to `aliveSessionSet` in `Env.registerSessionInfo`, preventing new session tracking until the memory leak is fixed. * Disabled the code that updates session report times in `Env.refreshSession`, pausing session activity tracking. Temporary table cleanup changes: * Commented out the logic in `TemporaryTableMgr.runAfterCatalogReady` that deletes temporary tables when their creating session is gone, halting automatic cleanup. ### Release note None ### Check List (For Author) - Test  - [ ] Regression test - [ ] Unit Test - [ ] Manual test (add detailed scripts or steps below) - [x] No need to test or manual test. Explain why: - [ ] This is a refactor/code format and no logic has been changed. - [x] Previous test can cover this change. - [ ] No code files have been changed. - [ ] Other reason  - Behavior changed: - [x] No. - [ ] Yes.  - Does this need documentation? - [x] No. - [ ] Yes.  ### Check List (For Reviewer who merge this PR) - [ ] Confirm the release note - [ ] Confirm test cases - [ ] Confirm document - [ ] Add branch pick label  (cherry picked from commit d172147) * [test](regression) Skip test_temp_table_with_restart after apache#59535 backport ### What problem does this PR solve? Related PR: apache#59535 Problem Summary: The backport of apache#59535 disables orphan temporary table auto-cleanup (TemporaryTableMgr) to stop the aliveSessionSet memory leak. test_temp_table_with_restart asserts exactly that auto-cleanup behavior and would always fail now. Skip it, mirroring upstream's practice in apache#64525. ### Release note None ### Check List (For Author) - Test: No need to test (test-only change, skipping a case) - Behavior changed: No - Does this need documentation: No * [fix](temp-table) Preserve alive-session reporting for mixed-version rolling upgrade ### What problem does this PR solve? Related PR: apache#59535 Problem Summary: apache#59535 disabled the leaking session-tracking bookkeeping (Env#aliveSessionSet / Env#sessionReportTimeMap) by commenting out registerSessionInfo/refreshSession, so aliveSessionSet always stays empty. During a rolling upgrade, an old-version master FE still queries follower FEs for their alive session ids (getAllAliveSessionIds RPC) and drops temporary tables whose owning session is not reported alive after loss_conn_fe_temp_table_keep_second (default 60s). A new-version follower would report an empty list, so the old master would wrongly delete temp tables of sessions that are actually alive on the new follower. Fix: derive getAllAliveSessionIds() from the live connection registry (ExecuteEnv -> ConnectScheduler -> getConnectionMap, which covers both the MySQL protocol pool and the Arrow Flight SQL pool) instead of the always-empty aliveSessionSet. The registry always reflects real client connections and needs no register/unregister bookkeeping, so alive-session reporting stays accurate for an old-version master while the leak fix of apache#59535 remains intact (registerSessionInfo/refreshSession/TemporaryTableMgr stay commented out). ### Release note None ### Check List (For Author) - Test: Unit Test (EnvTest#testGetAllAliveSessionIds), ./build.sh --fe - Behavior changed: No - Does this need documentation: No * [fix](temp-table) Register follower-created temp tables for cleanup on disconnect ### What problem does this PR solve? Related PR: apache#59535 Problem Summary: When a client connects to a follower FE and creates a temporary table, the CREATE statement is forwarded to master FE and executed there with a proxy ConnectContext (isProxy=true). Env.registerTempTableAndSession() therefore records the temp table's inner name only in the master-side proxy session, and the original client session's dbToTempTableNamesMap on the follower stays empty. When the follower session later disconnects, times out or is killed, ConnectContext.deleteTempTable() iterates an empty map and drops nothing. apache#59535 disabled the TemporaryTableMgr background cleanup that previously collected such orphan temp tables, so temp tables created via a follower leak permanently (unbounded metadata/tablet/storage leak). Fix: after a statement is successfully forwarded to master FE (StmtExecutor.forwardToMaster), if it is a CREATE TEMPORARY TABLE (nereids CreateTableCommand or legacy CreateTableStmt), register the deterministic inner name (sessionId + sign + tableName, same as master side since the session id is forwarded) into the follower session's dbToTempTableNamesMap; if it is a forwarded DROP TABLE, remove the entry symmetrically so a later disconnect does not re-drop it. Also fix the previously dead non-master branch of ConnectContext.deleteTempTable() to qualify the forwarded DROP with the db name recorded in the map, instead of relying on the session's current database at disconnect time. ### Release note Fix the issue that temporary tables created on a follower FE are not cleaned up after the session disconnects. ### Check List (For Author) - Test: Unit Test (existing org.apache.doris.catalog.EnvTest passed); added docker regression test temp_table_p0/test_temp_table_follower_disconnect (NOT run locally, docker regression environment unavailable) - Behavior changed: Yes - temporary tables created through a follower FE connection are now dropped when the session exits, as originally intended - Does this need documentation: No --------- Co-authored-by: morrySnow <zhangwenxin@selectdb.com>

[fix](temp-table) stop clean temp table until fix mem leak
a398de3

morrySnow requested review from CalvinKirs, dataroaring and morningman as code owners January 4, 2026 08:53

morrySnow added the dev/4.0.x label Jan 4, 2026

morrySnow added the p0_b label Jan 4, 2026

dataroaring approved these changes Jan 5, 2026
View reviewed changes

github-actionsBot added the approved Indicates a PR has been approved by one committer. label Jan 5, 2026

github-actionsBot added the reviewed label Jan 5, 2026

924060929 approved these changes Jan 5, 2026
View reviewed changes

dataroaring merged commit d172147 into apache:masterJan 5, 2026
31 of 32 checks passed

github-actionsBot mentioned this pull request Jan 5, 2026
branch-4.0: [fix](temp-table) not clean temp table temporary until fix mem leak #59535 #59551
Merged

morrySnow deleted the temp_fix_oom branch January 5, 2026 03:31

yiguolei pushed a commit that referenced this pull request Jan 7, 2026
branch-4.0: [fix](temp-table) not clean temp table temporary until fi…
82d28ef
…x mem leak #59535 (#59551) Cherry-picked from #59535 Co-authored-by: morrySnow <zhangwenxin@selectdb.com>

yiguolei added dev/4.0.3-merged and removed dev/4.0.x labels Jan 7, 2026

seawinde mentioned this pull request Jan 13, 2026
[draft](connectionContext) Fix potential memory leak by ensuring ConnectContext#cleanup() is called when ConnectionContext is no longer in use #59510
Closed
16 tasks

ybtsdst pushed a commit to ybtsdst/doris that referenced this pull request Feb 27, 2026
branch-4.0: [fix](temp-table) not clean temp table temporary until fi…
085dd13
…x mem leak apache#59535 (apache#59551) Cherry-picked from apache#59535 Co-authored-by: morrySnow <zhangwenxin@selectdb.com>

Ryan19929 mentioned this pull request Sep 1, 2026
[fix](temp-table) not clean temp table temporary until fix mem leak (#59535) HYDCP/hy-doris#93
Merged

Ryan19929 mentioned this pull request Sep 2, 2026
3.1.4-hydcp-1.3版本：Bug fixes and stability improvements HYDCP/hy-doris#64
Open

Navigation Menu

Uh oh!

[fix](temp-table) not clean temp table temporary until fix mem leak - #59535

[fix](temp-table) not clean temp table temporary until fix mem leak#59535
dataroaring merged 1 commit into
apache:masterfrom
morrySnow:temp_fix_oom

morrySnow commented Jan 4, 2026

Uh oh!

hello-stephen commented Jan 4, 2026

Uh oh!

morrySnow commented Jan 4, 2026

Uh oh!

doris-robot commented Jan 4, 2026

Uh oh!

doris-robot commented Jan 4, 2026

Uh oh!

doris-robot commented Jan 4, 2026

Uh oh!

hello-stephen commented Jan 4, 2026

Uh oh!

hello-stephen commented Jan 4, 2026

Uh oh!

dataroaring left a comment

Uh oh!

github-actionsBot commented Jan 5, 2026

Uh oh!

github-actionsBot commented Jan 5, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

Uh oh!

Conversation

morrySnow commented Jan 4, 2026

What problem does this PR solve?

Release note

Check List (For Author)

Check List (For Reviewer who merge this PR)

Uh oh!

hello-stephen commented Jan 4, 2026

Uh oh!

morrySnow commented Jan 4, 2026

Uh oh!

doris-robot commented Jan 4, 2026

Uh oh!

doris-robot commented Jan 4, 2026

Uh oh!

doris-robot commented Jan 4, 2026

Uh oh!

hello-stephen commented Jan 4, 2026

FE UT Coverage Report

Uh oh!

hello-stephen commented Jan 4, 2026

FE Regression Coverage Report

Uh oh!

dataroaring left a comment

Choose a reason for hiding this comment

Uh oh!

github-actionsBot commented Jan 5, 2026

Uh oh!

github-actionsBot commented Jan 5, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants