Uh oh!
There was an error while loading. Please reload this page.
branch-4.1: [Improve](streaming-job) use per-table publication instead of ALL TABLES for PostgreSQL CDC #62526 - #62793
Merged
Conversation
Thearas
commented
Apr 24, 2026
Contributor
Thank you for your contribution to Apache Doris. Please clearly describe your PR:
|
Thearas
commented
Apr 24, 2026
Contributor
run buildall |
JNSimba
commented
Apr 24, 2026
Member
run cloud_p0 |
1 similar comment
JNSimba
commented
Apr 24, 2026
Member
run cloud_p0 |
JNSimba
commented
Apr 24, 2026
Member
run nonConcurrent |
hello-stephen
commented
Apr 24, 2026
Contributor
FE Regression Coverage ReportIncrement line coverage |
…LES for PostgreSQL CDC (#62526) ### What problem does this PR solve? Problem Summary: Previously, PostgreSQL CDC streaming jobs created publications using `FOR ALL TABLES`, which monitors all tables in the database regardless of which tables the job actually needs to capture. This causes unnecessary WAL decoding overhead and network traffic. Additionally, users could not specify custom `slot_name` or `publication_name`, and these values were not visible in `SHOW JOBS`. This PR makes the following improvements: 1. **Per-table publication (FILTERED mode)**: Publication is created with `FOR TABLE table1, table2` instead of `FOR ALL TABLES`, only including tables specified in `include_tables` (or `table` for the cdc_stream TVF path). 2. **Configurable slot_name and publication_name**: Users can optionally pass `slot_name` and `publication_name` in source properties. If not specified, auto-generated names `doris_cdc_{jobId}` / `doris_pub_{jobId}` are used. 3. **Visible in SHOW (from-to path)**: For the `CREATE JOB ... FROM POSTGRES TO` path, auto-generated or user-provided names are persisted in `sourceProperties` and visible via `SHOW JOBS`. 4. **Partition table support for FILTERED mode**: Added `publish_via_partition_root = true` option for PostgreSQL 13+ in FILTERED mode (previously only in ALL_TABLES mode). 5. **Per-resource ownership cleanup**: Auto-created slot/publication (names matching `doris_cdc_{jobId}` / `doris_pub_{jobId}`) are dropped when the job is deleted; user-provided names are preserved. Each resource is evaluated independently, so a job with a user publication and an auto slot will only drop the auto slot. 6. **Validation scope**: PG slot/publication ownership validation runs at CREATE JOB time for both paths: - From-to job: validated against `sourceProperties` in `StreamingInsertJob.init`. - `cdc_stream` TVF wrapped inside a streaming INSERT job: validated in `StreamingInsertJob.initInsertJob` against a temporary copy of the TVF properties with defaults populated; the TVF properties map itself is read-only because Nereids may hand back an immutable map. - Standalone `cdc_stream` TVF (SELECT without an enclosing job): no PG resource validation — the TVF is a point query and not the unit that owns a slot/publication. Users supplying an invalid slot/publication will get the failure from cdcclient at execution time. 7. **Task-time default injection for the TVF path**: `JdbcTvfSourceOffsetProvider.ensureInitialized` and `rewriteTvfParams` populate default slot/publication names into `sourceProperties` and per-task TVF props so cdcclient ownership logic sees the resolved names both when the reader is created and when `/api/close` tears it down. Without the `ensureInitialized` step, `cleanMeta -> /api/close` would send a config without `publication_name`, cdcclient would fall back to the legacy `dbz_publication` on close, and DROP JOB would leave the Doris-managed publication behind.
morningman
commented
Apr 24, 2026
Contributor
run buildall |
morningmanforce-pushed
the
auto-pick-62526-branch-4.1
branch
from
April 24, 2026 20:22
0ded83c to
9faf3b4Comparehello-stephen
commented
Apr 24, 2026
Contributor
FE UT Coverage ReportIncrement line coverage |
hello-stephen
commented
Apr 24, 2026
Contributor
FE Regression Coverage ReportIncrement line coverage |
JNSimba
commented
Apr 24, 2026
Member
run p0 |
1 similar comment
JNSimba
commented
Apr 25, 2026
Member
run p0 |
hello-stephen
commented
Apr 25, 2026
Contributor
FE Regression Coverage ReportIncrement line coverage |
hello-stephen
commented
Apr 25, 2026
Contributor
FE Regression Coverage ReportIncrement line coverage |
yiguolei
approved these changes
Apr 27, 2026
Uh oh!
There was an error while loading. Please reload this page.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment
6 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Cherry-picked from #62526