[Improve](streaming job) support postgres partition table sync by JNSimba · Pull Request #60560 · apache/doris

JNSimba · 2026-02-06T02:05:22Z

What problem does this PR solve?

Related PR: #59461

To enhance partition table synchronization, add publish_via_partition_root when creating a PUBLICATION instance, specifically for PG 13+.

Release note

None

Check List (For Author)

Test
- Regression test
- Unit Test
- Manual test (add detailed scripts or steps below)
- No need to test or manual test. Explain why:
  - This is a refactor/code format and no logic has been changed.
  - Previous test can cover this change.
  - No code files have been changed.
  - Other reason
Behavior changed:
- No.
- Yes.
Does this need documentation?
- No.
- Yes.

Check List (For Reviewer who merge this PR)

Confirm the release note
Confirm test cases
Confirm document
Add branch pick label

Thearas · 2026-02-06T02:05:32Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

What problem was fixed (it's best to include specific error reporting information). How it was fixed.
Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
What features were added. Why was this function added?
Which code was refactored and why was this part of the code refactored?
Which functions were optimized and what is the difference before and after the optimization?

JNSimba · 2026-02-06T02:07:35Z

run buildall

Copilot

Pull request overview

This PR enhances PostgreSQL partition table synchronization in streaming jobs by adding support for publish_via_partition_root parameter when creating publication instances for PostgreSQL 13+. This is a follow-up to PR #59461 which added PostgreSQL synchronization support.

Changes:

Added support for PostgreSQL partition tables by enabling publish_via_partition_root in publications for PostgreSQL 13+
Enhanced database configuration to support flexible database specification (either in JDBC URL or as a separate parameter)
Added comprehensive regression test for partition table synchronization including snapshot, incremental changes, and dynamic partition creation

Reviewed changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 4 comments.

File	Description
test_streaming_postgres_job_partition.groovy	New regression test validating partition table synchronization with insert, update, delete operations and dynamic partition creation
test_streaming_postgres_job_partition.out	Expected output for the partition table test showing snapshot and binlog data
PostgresSourceReader.java	Enhanced database configuration flexibility and enabled partition table scanning via `setIncludePartitionedTables(true)`
PostgresReplicationConnection.java	Copied and modified Debezium file adding `publish_via_partition_root` parameter for PostgreSQL 13+ in ALL_TABLES publication mode

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

JNSimba · 2026-02-06T04:51:23Z

run buildall

doris-robot · 2026-02-06T05:59:22Z

TPC-H: Total hot run time: 30509 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit a348507eae8387a099abe8e4445562bf70b0f290, data reload: false
------ Round 1 ----------------------------------
q1	17608	4465	4285	4285
q2	2114	349	240	240
q3	10119	1382	743	743
q4	10273	811	319	319
q5	8943	2247	1975	1975
q6	229	181	148	148
q7	897	738	611	611
q8	9303	1447	1147	1147
q9	4640	4669	4638	4638
q10	6844	1936	1560	1560
q11	525	298	290	290
q12	332	381	219	219
q13	17783	4149	3256	3256
q14	233	236	216	216
q15	874	815	798	798
q16	674	679	621	621
q17	700	831	534	534
q18	6769	5854	5925	5854
q19	1236	988	619	619
q20	500	493	388	388
q21	2582	1871	1766	1766
q22	367	325	282	282
Total cold run time: 103545 ms
Total hot run time: 30509 ms
----- Round 2, with runtime_filter_mode=off -----
q1	4383	4334	4357	4334
q2	263	364	248	248
q3	2086	2728	2215	2215
q4	1356	1759	1302	1302
q5	4355	4125	4332	4125
q6	214	177	134	134
q7	1887	1804	1704	1704
q8	2496	2342	2270	2270
q9	7039	7052	6974	6974
q10	2637	2799	2443	2443
q11	535	457	461	457
q12	630	669	571	571
q13	3591	4092	3246	3246
q14	295	299	267	267
q15	839	810	807	807
q16	644	676	661	661
q17	1102	1316	1313	1313
q18	7716	7374	7180	7180
q19	841	849	834	834
q20	1948	2036	1846	1846
q21	4632	4210	4149	4149
q22	563	550	520	520
Total cold run time: 50052 ms
Total hot run time: 47600 ms

doris-robot · 2026-02-06T06:16:01Z

ClickBench: Total hot run time: 28.56 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit a348507eae8387a099abe8e4445562bf70b0f290, data reload: false
query1	0.05	0.05	0.05
query2	0.09	0.05	0.05
query3	0.25	0.08	0.08
query4	1.61	0.11	0.11
query5	0.28	0.26	0.26
query6	1.16	0.67	0.67
query7	0.04	0.03	0.02
query8	0.05	0.04	0.04
query9	0.56	0.50	0.49
query10	0.56	0.54	0.56
query11	0.14	0.10	0.10
query12	0.13	0.10	0.11
query13	0.62	0.60	0.62
query14	1.05	1.05	1.04
query15	0.87	0.85	0.88
query16	0.38	0.39	0.44
query17	1.12	1.15	1.15
query18	0.22	0.21	0.21
query19	2.03	2.00	2.02
query20	0.02	0.01	0.01
query21	15.42	0.26	0.14
query22	5.32	0.05	0.06
query23	16.26	0.28	0.11
query24	1.59	0.34	0.53
query25	0.07	0.10	0.08
query26	0.13	0.13	0.13
query27	0.07	0.05	0.05
query28	3.76	1.15	0.96
query29	12.54	3.90	3.20
query30	0.28	0.14	0.12
query31	2.84	0.61	0.40
query32	3.25	0.59	0.50
query33	3.27	3.20	3.35
query34	15.99	5.39	4.89
query35	4.83	4.80	4.75
query36	0.66	0.50	0.49
query37	0.11	0.07	0.07
query38	0.08	0.04	0.04
query39	0.04	0.03	0.03
query40	0.20	0.17	0.16
query41	0.09	0.04	0.03
query42	0.05	0.03	0.03
query43	0.05	0.03	0.03
Total cold run time: 98.13 s
Total hot run time: 28.56 s

JNSimba · 2026-02-06T07:15:21Z

run external

JNSimba · 2026-02-06T07:15:33Z

run vault_p0

JNSimba · 2026-02-06T07:46:09Z

run nonConcurrent

hello-stephen · 2026-02-06T10:03:38Z

FE Regression Coverage Report

Increment line coverage 0.00% (0/1) 🎉
Increment coverage report
Complete coverage report

github-actions · 2026-02-09T02:28:34Z

PR approved by at least one committer and no changes requested.

github-actions · 2026-02-09T02:28:37Z

PR approved by anyone and no changes requested.

### What problem does this PR solve? Related PR: #59461 To enhance partition table synchronization, add `publish_via_partition_root` when creating a PUBLICATION instance, specifically for PG 13+.

…e#60560) ### What problem does this PR solve? Related PR: apache#59461 To enhance partition table synchronization, add `publish_via_partition_root` when creating a PUBLICATION instance, specifically for PG 13+.

… sync #60560 (#60614) ### What problem does this PR solve? Cherry-picked from #60560, #60624 ### Release note None ### Check List (For Author) - Test  - [ ] Regression test - [ ] Unit Test - [ ] Manual test (add detailed scripts or steps below) - [ ] No need to test or manual test. Explain why: - [ ] This is a refactor/code format and no logic has been changed. - [ ] Previous test can cover this change. - [ ] No code files have been changed. - [ ] Other reason  - Behavior changed: - [ ] No. - [ ] Yes.  - Does this need documentation? - [ ] No. - [ ] Yes.  ### Check List (For Reviewer who merge this PR) - [ ] Confirm the release note - [ ] Confirm test cases - [ ] Confirm document - [ ] Add branch pick label

… sync apache#60560 (apache#60614) ### What problem does this PR solve? Cherry-picked from apache#60560, apache#60624 ### Release note None ### Check List (For Author) - Test  - [ ] Regression test - [ ] Unit Test - [ ] Manual test (add detailed scripts or steps below) - [ ] No need to test or manual test. Explain why: - [ ] This is a refactor/code format and no logic has been changed. - [ ] Previous test can cover this change. - [ ] No code files have been changed. - [ ] Other reason  - Behavior changed: - [ ] No. - [ ] Yes.  - Does this need documentation? - [ ] No. - [ ] Yes.  ### Check List (For Reviewer who merge this PR) - [ ] Confirm the release note - [ ] Confirm test cases - [ ] Confirm document - [ ] Add branch pick label

support Postgres partition tbl
6ba6724

fix
3638081

JNSimba requested a review from Copilot February 6, 2026 02:06

Copilotstarted reviewing on behalf of JNSimba February 6, 2026 02:07 View session

CopilotAI reviewed Feb 6, 2026
View reviewed changes

fix
a348507

JNSimba added the dev/4.0.x label Feb 9, 2026

liaoxin01 approved these changes Feb 9, 2026
View reviewed changes

github-actionsBot added approved Indicates a PR has been approved by one committer. reviewed labels Feb 9, 2026

CalvinKirs approved these changes Feb 9, 2026
View reviewed changes

JNSimba merged commit 7082e74 into apache:masterFeb 9, 2026
31 of 32 checks passed

github-actionsBot mentioned this pull request Feb 9, 2026
branch-4.0: [Improve](streaming job) support postgres partition table sync #60560 #60600
Closed

JNSimba mentioned this pull request Feb 9, 2026
branch-4.0: [Improve](streaming job) support postgres partition table sync #60560 #60614
Merged
16 tasks

yiguolei added dev/4.0.4-merged and removed dev/4.0.x labels Feb 24, 2026

yiguolei mentioned this pull request Mar 7, 2026
4.0.4 Release Notes #61121
Closed

Uh oh!

Conversation

JNSimba commented Feb 6, 2026• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What problem does this PR solve?

Release note

Check List (For Author)

Check List (For Reviewer who merge this PR)

Uh oh!

Thearas commented Feb 6, 2026

Uh oh!

JNSimba commented Feb 6, 2026

Uh oh!

CopilotAI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

JNSimba commented Feb 6, 2026

Uh oh!

doris-robot commented Feb 6, 2026

Uh oh!

doris-robot commented Feb 6, 2026

Uh oh!

JNSimba commented Feb 6, 2026

Uh oh!

JNSimba commented Feb 6, 2026

Uh oh!

JNSimba commented Feb 6, 2026

Uh oh!

hello-stephen commented Feb 6, 2026

FE Regression Coverage Report

Uh oh!

github-actionsBot commented Feb 9, 2026

Uh oh!

github-actionsBot commented Feb 9, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

8 participants

JNSimba commented Feb 6, 2026•
edited
Loading