Uh oh!
There was an error while loading. Please reload this page.
[regression-test](streaming-job) add cdc cases (composite/concurrent-dml/id-gap/decimal/datetime pk) and fix split-bound java.time deserialize - #63471
Conversation
…dml/id-gap/decimal/datetime pk) and fix split-bound java.time deserialize
JNSimba
commented
May 21, 2026
/review |
JNSimba
commented
May 21, 2026
run buildall |
There was a problem hiding this comment.
I found two issues that should be addressed before merging.
Critical checkpoint conclusions:
- Goal/test coverage: the PR adds CDC regression coverage and a java.time split-bound conversion fix, but one MySQL composite-PK test currently asserts the reversed key order and can mask a schema-mapping regression.
- Scope/focus: the implementation change is small; the test additions are broad but aligned with the PR goal.
- Concurrency/lifecycle: no new production concurrency was introduced; the modified pause/resume test still has a timing race in how it detects settled in-flight chunks.
- Compatibility/persistence: the split-bound conversion extends supported restored bound types; I did not find an additional blocking compatibility issue in that code.
- Parallel paths: MySQL and PostgreSQL paths are both covered, but their composite-PK order expectations are inconsistent.
- Test result correctness: new .out files match the new tests, but the MySQL composite-PK expected output records the reversed key order.
git diff --checkalso reports trailing whitespace/new blank EOF in several generated .out files. - Observability/performance/configuration: no new config or production hot-path performance issue found.
User focus: no additional user-provided review focus was specified.
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
…to avoid slow-chunk flake
JNSimba
commented
May 21, 2026
/review |
There was a problem hiding this comment.
Requesting changes because an existing blocking review thread is still unresolved: regression-test/suites/job_p0/streaming_job/cdc/test_streaming_mysql_job_composite_pk.groovy still asserts the reversed MySQL composite UNIQUE KEY order (order_id, tenant_id and role_id, user_id) instead of the source primary-key order. I did not add a duplicate inline comment because that exact issue is already covered by the existing thread.
Critical checkpoint conclusions:
- Goal/test proof: The PR aims to harden CDC snapshot split bound conversion and add regression coverage for date/time, decimal/unsigned, id gaps, composite PKs, concurrent DML, and pause/resume. Most added tests directly exercise those paths, but the MySQL composite-PK regression currently proves the wrong key order.
- Scope/focus: The implementation change is small and focused; most of the PR is targeted regression coverage.
- Concurrency/lifecycle: No new shared mutable concurrency or special lifecycle risk was found in the Java change; the pause/resume regression now uses a stronger settle window than the existing flagged version.
- Config/compatibility/protocol: No new config item, storage format, or FE-BE protocol compatibility issue found.
- Parallel paths: MySQL and PostgreSQL test paths were reviewed; the existing MySQL composite-key-order issue remains distinct from the PostgreSQL test, which expects source order.
- Tests/results: Generated
.outfiles exist and deterministic query ordering is used where result sets have multiple rows. The remaining blocker is that one generated result codifies the wrong MySQL composite key order. - Observability/performance/transactions: No additional observability, performance, transaction, or persistence issue found in the changed code.
User focus: No additional user-provided focus was specified.
JNSimba
commented
May 21, 2026
run buildall |
PR approved by at least one committer and no changes requested. |
Uh oh!
There was an error while loading. Please reload this page.
#63404#63471#63480#63490#63514#63618 (#63812) Cherry-picked from: - #63079 [improve](streaming-job) async chunk splitting for cdc source job - #63404 [test](streaming-job) refine cdc data-type and boundary regression cases for mysql/pg - #63471 [regression-test](streaming-job) add cdc cases (composite/concurrent-dml/id-gap/decimal/datetime pk) and fix split-bound java.time deserialize - #63480 [fix](streaming-job) misc fixes for typo/log/validation/visibility - #63402 [fix](streaming-job) drop neighbour-table rows leaked by JDBC LIKE wildcards in JdbcPostgreSQLClient - #63514 [regression-test](streaming-job) add cdc operational cases for offset modes and pg slot lifecycle - #63618 [fix](streaming-job) fix postgres historical-date timestamp handling in cdc-client - #63490 [improve](streaming-job) support user-specified mysql server_id with per-reader assignment
…dml/id-gap/decimal/datetime pk) and fix split-bound java.time deserialize (apache#63471) ### What problem does this PR solve? Extend CDC streaming-job regression coverage across MySQL/Postgres for primary-key related scenarios that the existing suites did not exercise; plus a small cdc-client follow-up to apache#63219 so split-bound restore handles the modern java.time types that some JDBC drivers return by default. #### Regression cases added | Case | Tables | Guarded by | |---|---|---| | `*_composite_pk` | composite PK + full-PK mapping table | composite PK chunk split + INSERT/UPDATE/DELETE locating across all PK columns; tenant/org-boundary chunk slicing | | `*_snapshot_with_concurrent_dml` | 1000-row table + unrelated decoy | source-side INSERT/UPDATE/DELETE during snapshot phase converges correctly; unrelated table outside include_tables not leaked | | `*_id_gap_completeness` | dense id 1~100 + outlier id 10000000 + post-outlier 10000001~10000100 | snapshot reader covers tail rows past the chunk-time max without dropping; binlog DML across all three id regions applies correctly | | `*_decimal_pk` | DECIMAL(20,4) PK + (MySQL only) BIGINT UNSIGNED PK | evenly-path BigDecimal / BigInteger arithmetic in chunk splitter; bigint unsigned values up to 2^64-1 | | \`mysql_*_datetime_pk\` | DATETIME(6) PK + composite (DATETIME, id) PK | java.time.LocalDateTime split-bound JSON round-trip; composite temporal PK locating |
…ne and TIMESTAMP/timestamptz pk (#63543) ### What problem does this PR solve? Add CDC streaming-job regression coverage for timezone behavior and TIMESTAMP / timestamptz chunk-key paths that the existing suites do not exercise. #### Cases added | Case | Tables | Guards | |---|---|---| | `*_source_timezone` (mysql + pg) | TIMESTAMP/timestamptz multi-precision + DATETIME/timestamp + DATE/date (pg: + `timetz`) | source-side multi-tz INSERT (+08 / -05 / UTC), NULL, UPDATE crossing tz, epoch boundary; pg `timetz` column kept as a regression guard for the upstream JVM-tz handling | | `*_jdbc_servertimezone` (mysql + pg) | TIMESTAMP/timestamptz + DATETIME/timestamp | recommended end-to-end config — align `jdbc_url`'s `serverTimezone`/`timezone` with Doris session `time_zone` (read at runtime so it works on any default tz) | | `*_timestamp_pk` (mysql + pg) | mysql TIMESTAMP(6) + composite (TIMESTAMP, id); pg timestamp(6) + timestamptz(6) + composite (timestamptz, id) | chunk-key split + INSERT/UPDATE/DELETE locating on TIMESTAMP/timestamptz PK; depends on #63471 for `LocalDateTime` / `OffsetDateTime` chunk-bound restore in `AbstractCdcSourceReader.convertBound` |
What problem does this PR solve?
Extend CDC streaming-job regression coverage across MySQL/Postgres for primary-key related scenarios that the existing suites did not exercise; plus a small cdc-client follow-up to #63219 so split-bound restore handles the modern java.time types that some JDBC drivers return by default.
Regression cases added
*_composite_pk*_snapshot_with_concurrent_dml*_id_gap_completeness100 + outlier id 10000000 + post-outlier 1000000110000100*_decimal_pkcdc-client follow-up to #63219
`AbstractCdcSourceReader.convertBound` was added in #63219 with `java.sql.Date/Time/Timestamp` valueOf branches but missed Java 8 types. MySQL Connector/J 8.x returns DATETIME columns as `LocalDateTime` by default, so its split-bound restore failed with:
```
Java 8 date/time type `java.time.LocalDateTime` not supported by default
```
Refactor into a static Map registry so new bound types are one-liners, and add `LocalDateTime` / `LocalDate` / `LocalTime` / `OffsetDateTime` entries using `parse`.
Release note
Add CDC streaming-job regression suites for composite PK / snapshot-time concurrent DML / id-gap completeness / DECIMAL+BIGINT UNSIGNED PK / DATETIME PK; fix CDC split-bound restore to support Java 8 date/time types.
Check List (For Author)
Test
Behavior changed:
Does this need documentation?