Skip to content

Kafka Connect: Update runtime distribution LICENSE and NOTICE - #18019

Open
rmoff wants to merge 7 commits into
apache:mainfrom
rmoff:kc-license-notice
Open

Kafka Connect: Update runtime distribution LICENSE and NOTICE#18019
rmoff wants to merge 7 commits into
apache:mainfrom
rmoff:kc-license-notice

Conversation

@rmoff

@rmoff rmoff commented Sep 8, 2026

Copy link
Copy Markdown
Contributor

#15212 is currently blocked on licenses, which this PR looks to fix.

This PR makes the Kafka Connect runtime distribution's LICENSE and NOTICE files account for the third-party code the artifact bundles, so the non-Hive distribution can go through a release. It continues the work @rdblue started in #16147: two of its commits (removing the Hive distribution, excluding jsr305) are taken from that PR with @rdblue's authorship preserved. Of #16147's five commits, its dependency commit is replaced here (for the reasons below) and its two runtime-deps.txt commits are superseded by regeneration.

What this changes

Dependencies

The runtime declares the same cloud SDK dependencies as the iceberg-aws-bundle, iceberg-gcp-bundle and iceberg-azure-bundle projects, copied from their build files, instead of a hand-picked subset. That gives the distribution the same feature set as the bundles (for example S3 Access Grants, the S3 analytics accelerator, Cloud KMS and Azure Key Vault) and lets the LICENSE and NOTICE content be kept in sync with the bundles' files. iceberg-bigquery stays in the distribution; the BigQuery Metastore catalog it provides is documented for this connector. The log4j binding is excluded, as the aws-bundle build does.

Why not depend on the bundle jars themselves

#16147 depended on the three bundle projects directly. I tried that shape, in both variants, and neither passes ./gradlew check:

  • Bundle jars plus their transitive dependencies (the Kafka Connect: Update distribution LICENSE and NOTICE #16147 shape): every cloud class ships twice, once shaded inside a bundle jar and once as a loose jar. checkClassUniqueness reports 170 colliding jar sets, about 47,000 duplicate classes, and lib/ is 351 MB. Both build-checks and kafka-connect-tests failed on this in CI, for Kafka Connect: Update distribution LICENSE and NOTICE #16147 as well as for the first revision of this PR.
  • Bundle jars only (transitive = false): 15 colliding sets remain and cannot be resolved from this build, because the bundles ship unrelocated copies of libraries that Hadoop and iceberg-core also need, at different versions (Woodstox 7.0 in the gcp bundle against 6.7 loose, three versions of commons-codec, Error Prone 2.10 against 2.49). Accepting them would need the first checkClassUniqueness baseline in the repository, and four shared libraries would diverge in version.

Mirroring the bundles' dependency lists produces exactly the same set of loose jars as the first variant minus the three fat jars: runtime-deps.txt is byte-identical, checkClassUniqueness passes with no baseline, and the zip shrinks from 337 MB to 193 MB. A Kafka Connect plugin directory is a directory of loose jars in any case; the bundle jars are built for the single-jar Spark and Flink deployments.

Inventory

runtime-deps.txt is regenerated with generateRuntimeDeps, following the same approach as the other bundles (for example gcp-bundle/runtime-deps.txt), and checkRuntimeDeps guards it. It lists 253 coordinates; the distribution contains 268 jars, 254 of them third-party.

LICENSE

Reconciled against the jars in the built zip, including code shaded inside other jars. Removed: the jsr305 entry and stale entries for jars no longer bundled (Nimbus, json-smart, Accessors Smart, ASM, JCIP, commons-beanutils, Google SDK platform for Java). Added, with full text for the non-Apache licences: Bouncy Castle (MIT); the MSV, RELAX NG Datatype (BSD) and isorelax (MIT) components shaded in Woodstox; FastDoubleParser and Schubfach (MIT) shaded in Jackson; the Mozilla Public Suffix List (MPL 2.0) shipped by HttpComponents and Guava; BoringSSL linked into grpc-netty-shaded and Conscrypt, and Tomcat Native into grpc-netty-shaded; the BSD-licensed Bloom filter code in Hadoop and the derivative works in Commons Math; and the Apache-licensed code shaded in Parquet, gRPC xDS, OpenTelemetry, Reactor Netty and the Azure SDK. Corrected: Google API Common, GAX and Auth Library are BSD 3-Clause, not Apache; Jakarta Activation is EDL 1.0, not EPL 2.0; JTS is listed under its EDL 1.0 option as in the Spark and Flink runtimes. Iceberg's own derived-code entries (Parquet, Kite, Presto, Delta Lake, Apache Commons, HttpComponents Client) are added as in the Spark runtime LICENSE.

NOTICE

Carries Iceberg's own Kite, Presto and Delta Lake attributions, which every bundled iceberg-* jar's NOTICE contains; the cloud notices from the bundle NOTICE files; and the substantive non-cloud notices (Hadoop's export control notice, Commons Math, Jakarta Activation, grpc-netty-shaded, Jackson including Schubfach, and Aircompressor as in the Spark runtime). Notices that contain only the standard Apache boilerplate are omitted, as they are covered by the top-level Apache Iceberg attribution.

Notes for reviewers

  • Removing the Hive distribution is user-facing. A follow-up can add it back with its own runtime-deps.txt if it is still wanted.

  • hadoop-shaded-guava and hadoop-shaded-protobuf have no entries of their own; their relocated contents (Guava, Protobuf, checker-qual, JSpecify) are covered by those entries, as gRPC and Netty cover grpc-netty-shaded.

  • Two entries were copied from gcp-bundle/LICENSE and then corrected, because that file is wrong: its RELAX NG Datatype block reproduces the Jakarta Activation licence text, and its isorelax block points at the unrelated jing-trang project. The RELAX NG text here is the BSD licence of the jing-trang datatype module that hosts this API, since the relaxngDatatype artifact itself ships no licence file. gcp-bundle/LICENSE should get the same fix separately.

  • Native libraries inside zstd-jni, aws-crt and conscrypt are listed under the wrapper's licence only, following the Flink runtime, aws-bundle and gcp-bundle respectively.

  • Against main, the third-party jar count grows from 240 to 254 and the zip from 184 MB to 193 MB, because the bundles' dependency lists carry more artifacts than the previous hand-picked set.

  • Some licensing items surfaced in review are pre-existing on main or are project-wide questions, and are left for maintainers rather than expanded into this PR: attribution for the Debezium-derived code in iceberg-kafka-connect-transforms (added in Kafka Connect: Add SMTs for Debezium and AWS DMS #11936); BSD-3 texts for the Snappy and Zstandard C libraries statically linked into snappy-java and zstd-jni, and the ISC/OpenSSL terms in AWS-LC inside aws-crt; the protoc-gen-validate notice carried via gRPC xDS; and a single stated rule for upstream-only notices. gcp-bundle/azure-bundle share several of these.

Verification

./gradlew :iceberg-kafka-connect:iceberg-kafka-connect-runtime:check passes: checkRuntimeDeps, checkClassUniqueness, distZip and the 13 Kafka Connect integration tests (REST catalog with S3 via MinIO) against the built distribution. LICENSE coverage was checked by opening every jar in the built zip, listing shaded packages, native libraries and embedded licence files, and reading each component's licence from its POM or embedded licence text; the bundle LICENSE and NOTICE files were block-compared against the result.


Prepared with Claude (mixture of Opus 4.8 and Fable 5.1), human-driven and reviewed.

rmoff and others added 4 commits September 10, 2026 11:06
Replace the hand-picked AWS, GCP and Azure SDK dependencies with the exact
dependency lists declared by the iceberg-aws-bundle, iceberg-gcp-bundle and
iceberg-azure-bundle projects, so the distribution offers the same features
as the bundles and its LICENSE and NOTICE content can be kept in sync with
theirs.

This supersedes "Use cloud provider bundles instead of direct deps"
(6c5f670 from apache#16147), which depended on the bundle projects directly.
That shape ships the shaded bundle jars alongside their own transitive
dependencies as loose jars, duplicating about 47,000 classes and failing
checkClassUniqueness. Depending on the bundle jars alone still leaves 15
duplicate sets at differing versions, because the bundles do not relocate
libraries such as Woodstox and commons-codec that Hadoop also needs.
Mirroring the lists yields the same set of loose jars with no duplicates.

Also excludes the log4j binding, as iceberg-aws-bundle does, and keeps
iceberg-bigquery, which the BigQuery Metastore catalog needs.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…pendencies

Regenerated via generateRuntimeDeps after replacing the hand-picked cloud SDK
dependencies with the dependency lists of the aws, gcp and azure bundles and
removing the Hive distribution.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
rmoff and others added 3 commits September 10, 2026 15:17
Accounts for the third parties bundled in the non-Hive runtime distribution,
checked jar by jar against the built zip, including code shaded inside other
jars. Removes the jsr305 entry (now excluded) and stale entries for jars that
are no longer bundled (Nimbus, json-smart, ASM, JCIP annotations,
commons-beanutils).

Adds entries for third parties that were bundled but unaccounted for, with
full licence text for the non-Apache ones: Bouncy Castle (MIT); the MSV,
RELAX NG Datatype (BSD) and isorelax (CDDL) components shaded in Woodstox;
FastDoubleParser and Schubfach (MIT) shaded in Jackson; the Mozilla Public
Suffix List (MPL 2.0) shipped by HttpComponents and Guava; BoringSSL and
Tomcat Native linked into grpc-netty-shaded and Conscrypt; the BSD-licensed
Bloom filter code in Hadoop and the derivative works in Commons Math; and the
Apache-licensed code shaded in Parquet (Thrift, fastutil), gRPC xDS (Envoy,
CEL, xDS, UDPA, PGV), OpenTelemetry, Reactor Netty and Azure SDK.

Corrects entries that named the wrong licence: Google API Common, GAX and
Auth Library are BSD 3-Clause; Jakarta Activation is EDL 1.0; JTS is listed
under its EDL 1.0 option as in the Spark and Flink runtimes. Restores the
copyright line in the ThreeTen Extra text, drops a duplicate JAXB entry, and
adds Iceberg's own derived-code entries (Parquet, Kite, Presto, Delta Lake,
Apache Commons, HttpComponents Client) as the Spark runtime LICENSE does.

Cloud entries are taken from the aws/gcp/azure bundle LICENSE files; where
those files were wrong (RELAX NG carried the Jakarta Activation text; the
Google libraries were correct there but not here) the text is taken from the
bundled jar or the upstream project instead.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Carries the required third-party attributions for the bundled jars: Iceberg's
own Kite, Presto and Delta Lake attributions, as in the NOTICE of every
bundled iceberg-* jar and in the Spark and Flink runtimes; the cloud notices
from the aws/gcp/azure bundle NOTICE files; and the substantive non-cloud
notices (Apache Hadoop export-control, Apache Commons Math, Jakarta
Activation, grpc-netty-shaded, Jackson including its Schubfach attribution,
and Airlift Aircompressor as in the Spark runtime). The AWS SDK block
reproduces the NOTICE shipped in the 2.54 jars in full. Boilerplate-only
Apache notices are omitted as covered by the top-level Apache Iceberg
attribution.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The Kafka Connect runtime now builds a single distribution rather than a
separate Hive-Metastore-client build. Update the docs to drop references to
choosing between two distributions and note that the Hive Metastore client
must be added to the plugin directory separately when using a Hive catalog.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@github-actions github-actions Bot added the docs label Sep 10, 2026
@rmoff
rmoff marked this pull request as ready for review September 10, 2026 18:55
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants