[Bug] Streaming Job

[297460389]

Search before asking

• I had searched in the issues and found no similar issues.

Version

doris 4.1.3

What's Wrong?

doris use mysql Streaming Job , fe.warn.log repoert WARN (EditLog-Flusher|76) [BDBJEJournal.write():172] write bdb journal batch data is too large, data size 714271, the first journal id 1517333, batch size 1

What You Expected?

How to eliminate this warning

How to Reproduce?

No response

Anything Else?

No response

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[297460389]

[Doris-Breakwater]

Initial triage (Doris 4.1.3; the issue currently has no labels):

Verified facts

• In 4.1.3, BDBJEJournal.write(JournalBatch) commits the BDBJE transaction before emitting this warning. The warning is triggered when the committed batch is larger than batch_edit_log_max_byte_size, whose default is 640 KiB (655,360 bytes). Therefore, the reported line by itself does not indicate a failed journal write or data loss. The reported record was 714,271 bytes, 58,911 bytes above that advisory threshold.
• batch size 1 is important: this was one oversized journal entity, not too many small entries grouped together. Reducing batch_edit_log_max_item_num cannot split that entity.
• A 4.1.3 MySQL Streaming Job can persist the complete JDBC offset-provider state inside an OP_UPDATE_SCHEDULER_JOB edit log. During snapshot processing that state grows with split progress and is persisted on offset commits.
• PR #66238, backported to branch-4.1 by PR #66539 after the 4.1.3 tag, specifically reduces this behavior: it rate-limits snapshot offset persistence, increases the default snapshot split size, and removes obsolete snapshot recovery state after entering the binlog phase.

Assessment

The warning is informational/advisory unless it is accompanied by BDBJE write errors, high journal-write latency, or follower replay lag. The most likely explanation is the known 4.1.3 snapshot-offset persistence behavior addressed by #66238/#66539, but this is not yet confirmed because the issue does not identify the operation type of journal 1517333, the job phase, or the warning frequency.

If journal 1517333 is OP_UPDATE_SCHEDULER_JOB (numeric opcode 451) and the warnings occur while the MySQL job is taking its initial snapshot, the preferred resolution is to upgrade to a 4.1 build/release containing #66539 and verify the result. If the warning continues during the incremental/binlog phase after that change, the remaining serialized payload may be intrinsically large (for example, schema metadata for many/wide source tables) and should be investigated separately.

I do not recommend increasing batch_edit_log_max_byte_size merely to hide the warning. That changes the batching threshold and can allow larger BDBJE transactions; it does not reduce the metadata being persisted. An isolated warning can be left alone if journal latency and FE replay remain healthy.

Information needed to confirm the cause

Please provide:

1. The exact build/commit identifier in addition to 4.1.3, and whether this is shared-nothing or compute-storage-decoupled deployment.
2. Warning timestamps/frequency and whether they occur during job creation, initial snapshot, snapshot-to-binlog transition, or steady incremental consumption.
3. A credential-redacted CREATE JOB statement, the number of synchronized source tables, approximate total column count, and any explicitly configured snapshot split size.
4. The nearby CDC-client commit offset ... tableSchemasSize=... log line and any BDBJE write-error/slow-write lines from the same interval. Please redact connection properties and credentials.
5. If safely available from verbose logging or an offline BDBJE metadata copy, only the OpType and Size for journal ID 1517333. Do not publish the journal Data field because a serialized job may contain sensitive connection properties.

Maintainer next step: first confirm opcode 451 and correlate the warning with snapshot commits. If confirmed, treat #66539 as the existing branch-4.1 remediation and determine the first released 4.1.x version that contains it; otherwise inspect the identified operation's serialized fields before classifying this as a Streaming Job bug.

Breakwater-GitHub-Analysis-Slot: slot_453c7dc7d079