[fix](streaming-job) Fix PG replication slot leak when streaming task is cancelled during pause/resume by JNSimba · Pull Request #62010 · apache/doris

JNSimba · 2026-04-01T10:47:53Z

What problem does this PR solve?

Problem Summary:

When a PostgreSQL CDC streaming job is paused and resumed, the PG replication slot
can be permanently leaked, causing all subsequent tasks to fail with:
replication slot "doris_cdc_xxx" is active for PID xxx

Root cause:

The CDC client reuses a single SourceReader instance per jobId (Env.getOrCreateReader).
When FE cancels a task (PAUSE), the BE HTTP connection is closed, but the CDC client's
buildStreamRecords thread may still be blocked in pollRecords (up to 15s timeout).
Before the old task finishes, the new task (after RESUME) arrives at the same CDC client
and calls prepareStreamSplit, which overwrites this.streamReader with a new Fetcher
without closing the old one. The old Debezium reader (holding the PG replication connection)
is leaked — its reference is lost, so finishSplitRecords in the old task's finally block
closes the new Fetcher instead, and the PG slot is never released.

From the logs, the slot remained occupied for 25+ minutes until the test timed out:
Failed to start replication stream at LSN{0/318EBC8}; when setting up multiple connectors
for the same database host, please make sure to use a distinct replication slot name for each.

Fix:

Close the previous stream/binlog reader before creating a new one in
prepareStreamSplit (PG) and prepareBinlogSplit (MySQL). This ensures the old
Debezium connection is properly released when a new task reuses the same SourceReader.

Release note

None

Check List (For Author)

Test
- Regression test
- Unit Test
- Manual test (add detailed scripts or steps below)
- No need to test or manual test. Explain why:
  - This is a refactor/code format and no logic has been changed.
  - Previous test can cover this change.
  - No code files have been changed.
  - Other reason
Behavior changed:
- No.
- Yes.
Does this need documentation?
- No.
- Yes.

Check List (For Reviewer who merge this PR)

Confirm the release note
Confirm test cases
Confirm document
Add branch pick label

Thearas · 2026-04-01T10:47:59Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

What problem was fixed (it's best to include specific error reporting information). How it was fixed.
Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
What features were added. Why was this function added?
Which code was refactored and why was this part of the code refactored?
Which functions were optimized and what is the difference before and after the optimization?

JNSimba · 2026-04-01T10:48:21Z

run buildall

JNSimba · 2026-04-01T11:52:56Z

run buildall

JNSimba · 2026-04-01T12:14:53Z

run buildall

JNSimba · 2026-04-01T12:19:09Z

run buildall

JNSimba · 2026-04-01T15:27:08Z

run nonConcurrent

JNSimba · 2026-04-01T22:13:50Z

run external

JNSimba · 2026-04-02T00:04:38Z

run external

JNSimba · 2026-04-02T01:02:05Z

run external

sollhui

LGTM

github-actions · 2026-04-02T02:50:52Z

PR approved by anyone and no changes requested.

liaoxin01

LGTM

github-actions · 2026-04-02T04:11:42Z

PR approved by at least one committer and no changes requested.

… is cancelled during pause/resume (#62010) ### What problem does this PR solve? Problem Summary: When a PostgreSQL CDC streaming job is paused and resumed, the PG replication slot can be permanently leaked, causing all subsequent tasks to fail with: `replication slot "doris_cdc_xxx" is active for PID xxx` **Root cause:** The CDC client reuses a single `SourceReader` instance per jobId (`Env.getOrCreateReader`). When FE cancels a task (PAUSE), the BE HTTP connection is closed, but the CDC client's `buildStreamRecords` thread may still be blocked in `pollRecords` (up to 15s timeout). Before the old task finishes, the new task (after RESUME) arrives at the same CDC client and calls `prepareStreamSplit`, which overwrites `this.streamReader` with a new Fetcher without closing the old one. The old Debezium reader (holding the PG replication connection) is leaked — its reference is lost, so `finishSplitRecords` in the old task's finally block closes the new Fetcher instead, and the PG slot is never released. From the logs, the slot remained occupied for 25+ minutes until the test timed out: Failed to start replication stream at LSN{0/318EBC8}; when setting up multiple connectors for the same database host, please make sure to use a distinct replication slot name for each. **Fix:** Close the previous stream/binlog reader before creating a new one in `prepareStreamSplit` (PG) and `prepareBinlogSplit` (MySQL). This ensures the old Debezium connection is properly released when a new task reuses the same SourceReader.

…reaming task is cancelled during pause/resume #62010 (#62033) Cherry-picked from #62010 Co-authored-by: wudi <wudi@selectdb.com>

… is cancelled during pause/resume (#62010) ### What problem does this PR solve? Problem Summary: When a PostgreSQL CDC streaming job is paused and resumed, the PG replication slot can be permanently leaked, causing all subsequent tasks to fail with: `replication slot "doris_cdc_xxx" is active for PID xxx` **Root cause:** The CDC client reuses a single `SourceReader` instance per jobId (`Env.getOrCreateReader`). When FE cancels a task (PAUSE), the BE HTTP connection is closed, but the CDC client's `buildStreamRecords` thread may still be blocked in `pollRecords` (up to 15s timeout). Before the old task finishes, the new task (after RESUME) arrives at the same CDC client and calls `prepareStreamSplit`, which overwrites `this.streamReader` with a new Fetcher without closing the old one. The old Debezium reader (holding the PG replication connection) is leaked — its reference is lost, so `finishSplitRecords` in the old task's finally block closes the new Fetcher instead, and the PG slot is never released. From the logs, the slot remained occupied for 25+ minutes until the test timed out: Failed to start replication stream at LSN{0/318EBC8}; when setting up multiple connectors for the same database host, please make sure to use a distinct replication slot name for each. **Fix:** Close the previous stream/binlog reader before creating a new one in `prepareStreamSplit` (PG) and `prepareBinlogSplit` (MySQL). This ensures the old Debezium connection is properly released when a new task reuses the same SourceReader.

…reaming task is cancelled during pause/resume #62010 (#62736) Cherry-picked from #62010 Co-authored-by: wudi <wudi@selectdb.com>

JNSimba added dev/4.1.x labels Apr 1, 2026

JNSimba added 2 commits April 1, 2026 20:18

fix
b1800b4

fix mysql cdc client
6997b0c

JNSimba force-pushed the fix_pg_msyql_pause_resume branch from 9bfc27d to 6997b0cCompare April 1, 2026 12:18

sollhui approved these changes Apr 2, 2026
View reviewed changes

github-actionsBot added the reviewed label Apr 2, 2026

liaoxin01 approved these changes Apr 2, 2026
View reviewed changes

github-actionsBot added the approved Indicates a PR has been approved by one committer. label Apr 2, 2026

JNSimba merged commit c6dea59 into apache:masterApr 2, 2026
29 of 30 checks passed

github-actionsBot mentioned this pull request Apr 2, 2026
branch-4.1: [fix](streaming-job) Fix PG replication slot leak when streaming task is cancelled during pause/resume #62010 #62033
Merged

yiguolei pushed a commit that referenced this pull request Apr 3, 2026
branch-4.1: [fix](streaming-job) Fix PG replication slot leak when st…
b8e113b
…reaming task is cancelled during pause/resume #62010 (#62033) Cherry-picked from #62010 Co-authored-by: wudi <wudi@selectdb.com>

yiguolei added dev/4.1.0-merged and removed dev/4.1.x labels Apr 3, 2026

mymeiyi mentioned this pull request Apr 21, 2026
[fix](wal) Fix WAL replay loading 0 rows due to columns being marked as missing #62658
Merged

JNSimba added the dev/4.0.x label Apr 23, 2026

github-actionsBot mentioned this pull request Apr 23, 2026
branch-4.0: [fix](streaming-job) Fix PG replication slot leak when streaming task is cancelled during pause/resume #62010 #62736
Merged

yiguolei pushed a commit that referenced this pull request May 9, 2026
branch-4.0: [fix](streaming-job) Fix PG replication slot leak when st…
2375fb4
…reaming task is cancelled during pause/resume #62010 (#62736) Cherry-picked from #62010 Co-authored-by: wudi <wudi@selectdb.com>

yiguolei added dev/4.0.6-merged and removed dev/4.0.x labels May 9, 2026

JNSimba mentioned this pull request May 11, 2026
[fix](streaming-job) start counting task max interval after the first record is received #63141
Merged
3 tasks

morningman mentioned this pull request Jun 3, 2026
4.0.6 Release Notes #64065
Closed

Uh oh!

Conversation

JNSimba commented Apr 1, 2026

What problem does this PR solve?

What problem does this PR solve?

Release note

Check List (For Author)

Check List (For Reviewer who merge this PR)

Uh oh!

Thearas commented Apr 1, 2026

Uh oh!

JNSimba commented Apr 1, 2026

Uh oh!

JNSimba commented Apr 1, 2026

Uh oh!

JNSimba commented Apr 1, 2026

Uh oh!

JNSimba commented Apr 1, 2026

Uh oh!

JNSimba commented Apr 1, 2026

Uh oh!

JNSimba commented Apr 1, 2026

Uh oh!

JNSimba commented Apr 2, 2026

Uh oh!

JNSimba commented Apr 2, 2026

Uh oh!

sollhui left a comment

Choose a reason for hiding this comment

Uh oh!

github-actionsBot commented Apr 2, 2026

Uh oh!

liaoxin01 left a comment

Choose a reason for hiding this comment

Uh oh!

github-actionsBot commented Apr 2, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants