Skip to content

branch-4.1: [Feature](streamjob) Streaming job support cdc_stream TVF #61826 - #61970

Merged
yiguolei merged 1 commit into
branch-4.1from
auto-pick-61826-branch-4.1
Apr 1, 2026
Merged

branch-4.1: [Feature](streamjob) Streaming job support cdc_stream TVF #61826#61970
yiguolei merged 1 commit into
branch-4.1from
auto-pick-61826-branch-4.1

Conversation

@github-actions

@github-actionsgithub-actionsBot commented Mar 31, 2026

Copy link
Copy Markdown
Contributor

Cherry-picked from #61826

@JNSimbaYour Name
[Feature](streamjob) Streaming job support cdc_stream TVF (#61826)
bfde0ce
### What problem does this PR solve?
Previously, streaming jobs only supported the non-TVF path (FROM Source
TO Database) for CDC ingestion. This PR adds support for streaming jobs
to drive CDC ingestion via the
cdc_stream Table-Valued Function (TVF), enabling a more flexible and
SQL-native approach.
```sql
CREATE JOB test_streaming_job_cdc_stream_postgres_name
ON STREAMING DO INSERT INTO db.table SELECT name, age FROM cdc_stream(
"type" = "postgres",
"jdbc_url" = "jdbc:postgresql://127.0.0.1:5432/postgres",
"driver_url" = "postgresql-42.5.0.jar",
"driver_class" = "org.postgresql.Driver",
"user" = "postgres",
"password" = "123456",
"database" = "postgres",
"schema" = "cdc_test",
"table" = "test_streaming_job_cdc_stream_postgres_src",
"offset" = "initial"
) ```
Core flow:
1. Snapshot phase: FE fetches split chunks from BE, persists them to the
meta table, then creates one StreamingInsertTask per split (or batch).
The TVF parameters are
rewritten per-task with meta (split boundary info), job.id, and task.id.
2. Binlog phase: After all snapshot splits are completed, transitions to
continuous binlog ingestion via the same cdc_stream TVF with binlog
offset as meta.
3. Offset commit (2PC safe): After fetchRecordStream completes on BE,
the actual end offset is stored in PipelineCoordinator.taskOffsetCache.
FE pulls this via brpc in
beforeCommitted and stores it in the txn attachment — ensuring the
offset is committed atomically with the data.
4. FE restart recovery: State is recovered via txn replay
(replayOnCommitted) rather than EditLog, rebuilding
chunkHighWatermarkMap and remainingSplits from the meta table.
@hello-stephen

hello-stephen commented Mar 31, 2026

Copy link
Copy Markdown
Contributor

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@hello-stephen

hello-stephen commented Mar 31, 2026

Copy link
Copy Markdown
Contributor

run buildall

@JNSimba

JNSimba commented Mar 31, 2026

Copy link
Copy Markdown
Member

run nonConcurrent

@yiguolei
yiguolei merged commit fe86029 into branch-4.1Apr 1, 2026
26 of 29 checks passed
@morningman
morningman deleted the auto-pick-61826-branch-4.1 branch July 8, 2026 14:09
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@yiguoleiyiguoleiyiguolei approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@hello-stephen@JNSimba@yiguolei@dataroaring