Skip to content

Kafka Connect: validThroughTs() includes stale commitId entries from zombie coordinators #17077

Description

@thswlsqls

Apache Iceberg version
main @ 49b89a8

Query engine
N/A — Kafka Connect sink, engine-agnostic

Please describe the bug
CommitState.validThroughTs() (kafka-connect/kafka-connect/src/main/java/org/apache/iceberg/connect/channel/CommitState.java line 146) computes the VALID_THROUGH_TS_SNAPSHOT_PROP snapshot property from readyBuffer without filtering by commitId. addReady() (line 61) buffers every DATA_COMPLETE event regardless of commitId, including ones from a zombie coordinator still emitting events for a previous commit. As a result, a zombie coordinator's stale (or null-timestamp) partition offsets can pull the valid-through timestamp into the past or suppress it entirely (null), even though the current commit's data is complete and valid.

PR #16453 ("Kafka Connect: Make CommitState.isCommitReady() O(1)", merged) added the same Objects.equals(currentCommitId, payload.commitId()) filter to receivedPartitionCount and covered it with testIsCommitReadyIgnoresZombieCoordinatorPayloads, but validThroughTs() was left unfiltered — an incomplete fix.

Steps to reproduce

  1. Start a commit (startNewCommit()).
  2. Have a zombie coordinator emit DATA_COMPLETE with a different commitId (addReady()).
  3. Have the current coordinator emit its own DATA_COMPLETE.
  4. Call validThroughTs(false) — the zombie's offsets are still included in the min-timestamp calculation.

Additional context
N/A — reproducible via a pure unit test on CommitState, no Docker required.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions