[Feature] MySQL CDC: support regex (or wildcard) in exclude_tables / include_tables

[xiangge1208]

Search before asking

• I had searched in the issues and found no similar issues.

Description

I would like to request adding regular expression (or wildcard) support for the exclude_tables and include_tables configurations in CREATE JOB ... ON STREAMING FROM MYSQL.

Currently, exclude_tables / include_tables perform exact string matching only:
fe/fe-core/src/main/java/org/apache/doris/job/util/StreamingJobUtils.java

excludeTablesList = Arrays.asList(excludeTables.split(",")); // L378-380
// ...
if (includeTablesList.isEmpty() && !excludeTablesList.isEmpty() && excludeTablesList.contains(table)) { // L401-402
 continue;
}

Solution

Support regex (or at minimum glob-style wildcards) in both exclude_tables and
include_tables. Flink CDC's tables option already accepts regex
(e.g. app_db.\.*), so this would also align the two code paths.

Suggested semantics:

• Keep backward compatibility: a pattern with no regex metacharacters behaves exactly as today.
• Match against the bare table name (as today), not the qualified name, to avoid ambiguity.
• Document whether the match is full-match or partial-match (matches() vs find()) —
  full-match is less surprising.

Two related nits found while investigating, worth fixing in the same change:

1. No trimming. excludeTables.split(",") keeps surrounding whitespace, so
   "a, b" silently fails to exclude b. A .trim() per element would prevent a
   class of hard-to-spot config errors.
2. Error ordering. generateCreateTableCmds collects noPrimaryKeyTables while
   iterating, but throws only after the loop
   (StreamingJobUtils.java L469-472). Tables processed before the failure have already
   been created in Doris, so a failed CREATE JOB leaves partial state that must be
   cleaned up manually. Failing fast, or rolling back created tables, would be friendlier.

Use case

For whole-database sync this becomes hard to maintain. In our production database
(285 base tables) there are 16+ manually-created backup/archive tables following
recognizable naming patterns:

sys_config_bak_20260730
sys_config_bak_20260803_codex
sys_config_bak_20260804
sys_config_bak_codex_progress_20260810
sys_config_bak_mcp_20260812
sys_config_bak_progress_20260813
sys_config_bak_progress_202608132
sys_config_bak_xie_20260806
sys_config_backup_20260710_codex_mcp_skill_sync_fix
sys_config_backup_20260710_codex_mcp_workflow
plugin_config_backup_20260721
sys_credit_record_archive_bak_20260805
sys_dept_permission_scope_bak_20260509
sys_permission_bak_20260808
sys_permission_model_bak_20260820
sys_role_bak_20260808

With exact matching, every one of these must be enumerated, and — more importantly —
new backup tables created upstream after the job was created are silently picked up.
Since exclude_tables cannot be changed by ALTER JOB, keeping the exclusion list current
requires dropping and recreating the job (and carefully preserving the binlog offset).

A pattern like sys_config_bak_.* or *_bak_* would express the intent once and stay correct.

Related issues

No response

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct