Kafka Connect: Update runtime distribution LICENSE and NOTICE - #18019
Open
rmoff wants to merge 7 commits into
Open
Kafka Connect: Update runtime distribution LICENSE and NOTICE#18019rmoff wants to merge 7 commits into
rmoff wants to merge 7 commits into
Conversation
Replace the hand-picked AWS, GCP and Azure SDK dependencies with the exact dependency lists declared by the iceberg-aws-bundle, iceberg-gcp-bundle and iceberg-azure-bundle projects, so the distribution offers the same features as the bundles and its LICENSE and NOTICE content can be kept in sync with theirs. This supersedes "Use cloud provider bundles instead of direct deps" (6c5f670 from apache#16147), which depended on the bundle projects directly. That shape ships the shaded bundle jars alongside their own transitive dependencies as loose jars, duplicating about 47,000 classes and failing checkClassUniqueness. Depending on the bundle jars alone still leaves 15 duplicate sets at differing versions, because the bundles do not relocate libraries such as Woodstox and commons-codec that Hadoop also needs. Mirroring the lists yields the same set of loose jars with no duplicates. Also excludes the log4j binding, as iceberg-aws-bundle does, and keeps iceberg-bigquery, which the BigQuery Metastore catalog needs. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
(cherry picked from commit 48a730d)
(cherry picked from commit b7a3f89)
…pendencies Regenerated via generateRuntimeDeps after replacing the hand-picked cloud SDK dependencies with the dependency lists of the aws, gcp and azure bundles and removing the Hive distribution. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
rmoff
force-pushed
the
kc-license-notice
branch
from
September 10, 2026 10:52
eb1c338 to
f055eed
Compare
Accounts for the third parties bundled in the non-Hive runtime distribution, checked jar by jar against the built zip, including code shaded inside other jars. Removes the jsr305 entry (now excluded) and stale entries for jars that are no longer bundled (Nimbus, json-smart, ASM, JCIP annotations, commons-beanutils). Adds entries for third parties that were bundled but unaccounted for, with full licence text for the non-Apache ones: Bouncy Castle (MIT); the MSV, RELAX NG Datatype (BSD) and isorelax (CDDL) components shaded in Woodstox; FastDoubleParser and Schubfach (MIT) shaded in Jackson; the Mozilla Public Suffix List (MPL 2.0) shipped by HttpComponents and Guava; BoringSSL and Tomcat Native linked into grpc-netty-shaded and Conscrypt; the BSD-licensed Bloom filter code in Hadoop and the derivative works in Commons Math; and the Apache-licensed code shaded in Parquet (Thrift, fastutil), gRPC xDS (Envoy, CEL, xDS, UDPA, PGV), OpenTelemetry, Reactor Netty and Azure SDK. Corrects entries that named the wrong licence: Google API Common, GAX and Auth Library are BSD 3-Clause; Jakarta Activation is EDL 1.0; JTS is listed under its EDL 1.0 option as in the Spark and Flink runtimes. Restores the copyright line in the ThreeTen Extra text, drops a duplicate JAXB entry, and adds Iceberg's own derived-code entries (Parquet, Kite, Presto, Delta Lake, Apache Commons, HttpComponents Client) as the Spark runtime LICENSE does. Cloud entries are taken from the aws/gcp/azure bundle LICENSE files; where those files were wrong (RELAX NG carried the Jakarta Activation text; the Google libraries were correct there but not here) the text is taken from the bundled jar or the upstream project instead. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Carries the required third-party attributions for the bundled jars: Iceberg's own Kite, Presto and Delta Lake attributions, as in the NOTICE of every bundled iceberg-* jar and in the Spark and Flink runtimes; the cloud notices from the aws/gcp/azure bundle NOTICE files; and the substantive non-cloud notices (Apache Hadoop export-control, Apache Commons Math, Jakarta Activation, grpc-netty-shaded, Jackson including its Schubfach attribution, and Airlift Aircompressor as in the Spark runtime). The AWS SDK block reproduces the NOTICE shipped in the 2.54 jars in full. Boilerplate-only Apache notices are omitted as covered by the top-level Apache Iceberg attribution. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The Kafka Connect runtime now builds a single distribution rather than a separate Hive-Metastore-client build. Update the docs to drop references to choosing between two distributions and note that the Hive Metastore client must be added to the plugin directory separately when using a Hive catalog. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
rmoff
force-pushed
the
kc-license-notice
branch
from
September 10, 2026 14:22
f055eed to
b0651a2
Compare
rmoff
marked this pull request as ready for review
September 10, 2026 18:55
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
#15212 is currently blocked on licenses, which this PR looks to fix.
This PR makes the Kafka Connect runtime distribution's
LICENSEandNOTICEfiles account for the third-party code the artifact bundles, so the non-Hive distribution can go through a release. It continues the work @rdblue started in #16147: two of its commits (removing the Hive distribution, excluding jsr305) are taken from that PR with @rdblue's authorship preserved. Of #16147's five commits, its dependency commit is replaced here (for the reasons below) and its tworuntime-deps.txtcommits are superseded by regeneration.What this changes
Dependencies
The runtime declares the same cloud SDK dependencies as the
iceberg-aws-bundle,iceberg-gcp-bundleandiceberg-azure-bundleprojects, copied from their build files, instead of a hand-picked subset. That gives the distribution the same feature set as the bundles (for example S3 Access Grants, the S3 analytics accelerator, Cloud KMS and Azure Key Vault) and lets the LICENSE and NOTICE content be kept in sync with the bundles' files.iceberg-bigquerystays in the distribution; the BigQuery Metastore catalog it provides is documented for this connector. The log4j binding is excluded, as the aws-bundle build does.Why not depend on the bundle jars themselves
#16147 depended on the three bundle projects directly. I tried that shape, in both variants, and neither passes
./gradlew check:checkClassUniquenessreports 170 colliding jar sets, about 47,000 duplicate classes, andlib/is 351 MB. Bothbuild-checksandkafka-connect-testsfailed on this in CI, for Kafka Connect: Update distribution LICENSE and NOTICE #16147 as well as for the first revision of this PR.transitive = false): 15 colliding sets remain and cannot be resolved from this build, because the bundles ship unrelocated copies of libraries that Hadoop and iceberg-core also need, at different versions (Woodstox 7.0 in the gcp bundle against 6.7 loose, three versions of commons-codec, Error Prone 2.10 against 2.49). Accepting them would need the firstcheckClassUniquenessbaseline in the repository, and four shared libraries would diverge in version.Mirroring the bundles' dependency lists produces exactly the same set of loose jars as the first variant minus the three fat jars:
runtime-deps.txtis byte-identical,checkClassUniquenesspasses with no baseline, and the zip shrinks from 337 MB to 193 MB. A Kafka Connect plugin directory is a directory of loose jars in any case; the bundle jars are built for the single-jar Spark and Flink deployments.Inventory
runtime-deps.txtis regenerated withgenerateRuntimeDeps, following the same approach as the other bundles (for examplegcp-bundle/runtime-deps.txt), andcheckRuntimeDepsguards it. It lists 253 coordinates; the distribution contains 268 jars, 254 of them third-party.LICENSE
Reconciled against the jars in the built zip, including code shaded inside other jars. Removed: the
jsr305entry and stale entries for jars no longer bundled (Nimbus, json-smart, Accessors Smart, ASM, JCIP, commons-beanutils, Google SDK platform for Java). Added, with full text for the non-Apache licences: Bouncy Castle (MIT); the MSV, RELAX NG Datatype (BSD) and isorelax (MIT) components shaded in Woodstox; FastDoubleParser and Schubfach (MIT) shaded in Jackson; the Mozilla Public Suffix List (MPL 2.0) shipped by HttpComponents and Guava; BoringSSL linked into grpc-netty-shaded and Conscrypt, and Tomcat Native into grpc-netty-shaded; the BSD-licensed Bloom filter code in Hadoop and the derivative works in Commons Math; and the Apache-licensed code shaded in Parquet, gRPC xDS, OpenTelemetry, Reactor Netty and the Azure SDK. Corrected: Google API Common, GAX and Auth Library are BSD 3-Clause, not Apache; Jakarta Activation is EDL 1.0, not EPL 2.0; JTS is listed under its EDL 1.0 option as in the Spark and Flink runtimes. Iceberg's own derived-code entries (Parquet, Kite, Presto, Delta Lake, Apache Commons, HttpComponents Client) are added as in the Spark runtime LICENSE.NOTICE
Carries Iceberg's own Kite, Presto and Delta Lake attributions, which every bundled
iceberg-*jar's NOTICE contains; the cloud notices from the bundle NOTICE files; and the substantive non-cloud notices (Hadoop's export control notice, Commons Math, Jakarta Activation, grpc-netty-shaded, Jackson including Schubfach, and Aircompressor as in the Spark runtime). Notices that contain only the standard Apache boilerplate are omitted, as they are covered by the top-level Apache Iceberg attribution.Notes for reviewers
Removing the Hive distribution is user-facing. A follow-up can add it back with its own
runtime-deps.txtif it is still wanted.hadoop-shaded-guavaandhadoop-shaded-protobufhave no entries of their own; their relocated contents (Guava, Protobuf, checker-qual, JSpecify) are covered by those entries, as gRPC and Netty covergrpc-netty-shaded.Two entries were copied from
gcp-bundle/LICENSEand then corrected, because that file is wrong: its RELAX NG Datatype block reproduces the Jakarta Activation licence text, and its isorelax block points at the unrelated jing-trang project. The RELAX NG text here is the BSD licence of the jing-trang datatype module that hosts this API, since therelaxngDatatypeartifact itself ships no licence file.gcp-bundle/LICENSEshould get the same fix separately.Native libraries inside
zstd-jni,aws-crtandconscryptare listed under the wrapper's licence only, following the Flink runtime, aws-bundle and gcp-bundle respectively.Against
main, the third-party jar count grows from 240 to 254 and the zip from 184 MB to 193 MB, because the bundles' dependency lists carry more artifacts than the previous hand-picked set.Some licensing items surfaced in review are pre-existing on
mainor are project-wide questions, and are left for maintainers rather than expanded into this PR: attribution for the Debezium-derived code iniceberg-kafka-connect-transforms(added in Kafka Connect: Add SMTs for Debezium and AWS DMS #11936); BSD-3 texts for the Snappy and Zstandard C libraries statically linked intosnappy-javaandzstd-jni, and the ISC/OpenSSL terms in AWS-LC insideaws-crt; the protoc-gen-validate notice carried via gRPC xDS; and a single stated rule for upstream-only notices.gcp-bundle/azure-bundleshare several of these.Verification
./gradlew :iceberg-kafka-connect:iceberg-kafka-connect-runtime:checkpasses:checkRuntimeDeps,checkClassUniqueness,distZipand the 13 Kafka Connect integration tests (REST catalog with S3 via MinIO) against the built distribution. LICENSE coverage was checked by opening every jar in the built zip, listing shaded packages, native libraries and embedded licence files, and reading each component's licence from its POM or embedded licence text; the bundle LICENSE and NOTICE files were block-compared against the result.Prepared with Claude (mixture of Opus 4.8 and Fable 5.1), human-driven and reviewed.