Uh oh!
There was an error while loading. Please reload this page.
SOLR-18300 : Update apache.kafka to V4 - #4610
Conversation
…he-_-solr into renovate/major-apache.kafka
| for (TopicPartition topicPartition : topicPartitions) { | ||
| if (topicPartition.topic().equals(topic)) { | ||
| long endOffset = consumer.position(topicPartition); | ||
| long committedOffset = consumer.committed(topicPartition).offset(); |
There was a problem hiding this comment.
https://github.com/apache/solr/actions/runs/28428066221/job/84235821648 , Simply updating the version was giving compilation error.
Evidences for supporting this change:
There was a problem hiding this comment.
I guess this is one of those deprecated apis that I saw now being fixed!
| apache-httpcomponents-httpcore = { module = "org.apache.httpcomponents:httpcore", version.ref = "apache-httpcomponents-httpcore" } | ||
| apache-httpcomponents-httpmime = { module = "org.apache.httpcomponents:httpmime", version.ref = "apache-httpcomponents-httpmime" } | ||
| apache-kafka-clients = { module = "org.apache.kafka:kafka-clients", version.ref = "apache-kafka" } | ||
| apache-kafka-kafka213 = { module = "org.apache.kafka:kafka_2.13", version.ref = "apache-kafka" } |
| // Global override for local integration testing | ||
| grant { | ||
| permission java.security.AllPermission; | ||
| }; | ||
There was a problem hiding this comment.
[pending-to-discuss] with reviewer on the right way here. This works, but this is surely not the right way.
This is required to resolve this error, which comes in the cross-dc-manager test which interacts with the kafka container which runs on docker daemon.
There was a problem hiding this comment.
I believe there are ways to get around this with a more narrow line in solr-tests.policy.
| swagger3-jaxrs2-jakarta = { module = "io.swagger.core.v3:swagger-jaxrs2-jakarta", version.ref = "swagger3" } | ||
| tdunning-tdigest = { module = "com.tdunning:t-digest", version.ref = "tdunning-tdigest" } | ||
| testcontainers = { module = "org.testcontainers:testcontainers", version.ref = "testcontainers" } | ||
| testcontainers-kafka = { module = "org.testcontainers:testcontainers-kafka", version.ref = "testcontainers" } |
There was a problem hiding this comment.
Since Kafka 4.x fully supports lightweight, fast KRaft startups, the cleanest modern approach is to spin up an ephemeral Docker container using Testcontainers. It completely isolates Kafka from the JVM running Solr. [Industry standards]
There was a problem hiding this comment.
Makes sense. We do this for Tika testing...
markrmiller
commented
Jul 4, 2026
Just curious: what motivated the upgrade? |
jaykay12
commented
Jul 5, 2026
I came across this thread: https://lists.apache.org/thread/fkb9dgtybbjz4f5jzk6qb6ysh18bt5n5 by @janhoy asking for help in getting upgrades done & I landed upon the PR: #3274 that was raised by solrbot initially some 1 year back & was still pending. I thought of taking it to closure. Since there were major changes that were required, I created Jira & started working on it. |
There was a problem hiding this comment.
Pull request overview
Upgrades CrossDC to Apache Kafka 4.3.1 and migrates integration tests from embedded Kafka to Testcontainers.
Changes:
- Updates Kafka dependencies, locks, licenses, documentation, and changelog.
- Adds a reusable Kafka Testcontainers rule and AdminClient-based topic creation.
- Adapts Kafka 4 APIs and isolates per-test topics.
Reviewed changes
Copilot reviewed 60 out of 60 changed files in this pull request and generated 4 comments.
Show a summary per file
| File | Description |
|---|---|
solr/solr-ref-guide/modules/deployment-guide/pages/cross-dc-replication.adoc | Documents Kafka compatibility. |
solr/modules/cross-dc/gradle.lockfile | Locks Kafka 4 dependencies. |
solr/licenses/zstd-jni-1.5.6-4.jar.sha1 | Removes obsolete checksum. |
solr/licenses/zstd-jni-1.5.6-10.jar.sha1 | Adds updated checksum. |
solr/licenses/testcontainers-kafka-2.0.5.jar.sha1 | Adds Testcontainers Kafka checksum. |
solr/licenses/scala-reflect-2.13.15.jar.sha1 | Removes unused checksum. |
solr/licenses/scala-logging_2.13-3.9.5.jar.sha1 | Removes unused checksum. |
solr/licenses/scala-library-2.13.18.jar.sha1 | Removes unused checksum. |
solr/licenses/scala-java8-compat_2.13-1.0.2.jar.sha1 | Removes unused checksum. |
solr/licenses/scala-collection-compat_2.13-2.10.0.jar.sha1 | Removes unused checksum. |
solr/licenses/rocksdbjni-7.9.2.jar.sha1 | Removes old checksum. |
solr/licenses/rocksdbjni-10.1.3.jar.sha1 | Adds updated checksum. |
solr/licenses/pcollections-4.0.1.jar.sha1 | Removes unused checksum. |
solr/licenses/paranamer-2.8.3.jar.sha1 | Removes unused checksum. |
solr/licenses/metrics-core-2.2.0.jar.sha1 | Removes unused checksum. |
solr/licenses/lz4-java-1.10.2.jar.sha1 | Adds updated checksum. |
solr/licenses/lz4-java-1.10.1.jar.sha1 | Removes old checksum. |
solr/licenses/kafka-transaction-coordinator-3.9.2.jar.sha1 | Removes Kafka 3 artifact checksum. |
solr/licenses/kafka-tools-api-3.9.2.jar.sha1 | Removes Kafka 3 artifact checksum. |
solr/licenses/kafka-streams-4.3.1.jar.sha1 | Adds Kafka Streams checksum. |
solr/licenses/kafka-streams-4.3.1-test.jar.sha1 | Adds Kafka test checksum. |
solr/licenses/kafka-streams-3.9.2.jar.sha1 | Removes old checksum. |
solr/licenses/kafka-streams-3.9.2-test.jar.sha1 | Removes old test checksum. |
solr/licenses/kafka-storage-api-3.9.2.jar.sha1 | Removes Kafka 3 artifact checksum. |
solr/licenses/kafka-storage-3.9.2.jar.sha1 | Removes Kafka 3 artifact checksum. |
solr/licenses/kafka-server-common-3.9.2.jar.sha1 | Removes Kafka 3 artifact checksum. |
solr/licenses/kafka-server-common-3.9.2-test.jar.sha1 | Removes Kafka 3 test checksum. |
solr/licenses/kafka-server-3.9.2.jar.sha1 | Removes Kafka 3 artifact checksum. |
solr/licenses/kafka-raft-3.9.2.jar.sha1 | Removes Kafka 3 artifact checksum. |
solr/licenses/kafka-metadata-3.9.2.jar.sha1 | Removes Kafka 3 artifact checksum. |
solr/licenses/kafka-group-coordinator-api-3.9.2.jar.sha1 | Removes Kafka 3 artifact checksum. |
solr/licenses/kafka-group-coordinator-3.9.2.jar.sha1 | Removes Kafka 3 artifact checksum. |
solr/licenses/kafka-clients-4.3.1.jar.sha1 | Adds Kafka client checksum. |
solr/licenses/kafka-clients-4.3.1-test.jar.sha1 | Adds Kafka client test checksum. |
solr/licenses/kafka-clients-3.9.2.jar.sha1 | Removes old checksum. |
solr/licenses/kafka-clients-3.9.2-test.jar.sha1 | Removes old test checksum. |
solr/licenses/kafka_2.13-NOTICE.txt | Removes obsolete server-bundle notice. |
solr/licenses/kafka_2.13-LICENSE-ASL.txt | Removes obsolete server-bundle license. |
solr/licenses/kafka_2.13-3.9.2.jar.sha1 | Removes server artifact checksum. |
solr/licenses/kafka_2.13-3.9.2-test.jar.sha1 | Removes server test checksum. |
solr/licenses/jackson-module-scala_2.13-2.22.0.jar.sha1 | Removes unused checksum. |
solr/licenses/jackson-dataformat-csv-2.22.0.jar.sha1 | Removes unused checksum. |
solr/licenses/commons-validator-1.10.1.jar.sha1 | Removes unused checksum. |
solr/licenses/commons-digester-2.1.jar.sha1 | Removes unused checksum. |
solr/licenses/commons-collections-3.2.2.jar.sha1 | Removes unused checksum. |
solr/licenses/commons-beanutils-1.11.0.jar.sha1 | Removes unused checksum. |
solr/licenses/argparse4j-0.7.0.jar.sha1 | Removes unused checksum. |
solr/cross-dc-manager/src/test/org/apache/solr/crossdc/manager/ZkConfigIntegrationTest.java | Migrates to containerized Kafka. |
solr/cross-dc-manager/src/test/org/apache/solr/crossdc/manager/SolrAndKafkaReindexTest.java | Migrates reindex testing. |
solr/cross-dc-manager/src/test/org/apache/solr/crossdc/manager/SolrAndKafkaMultiCollectionIntegrationTest.java | Adds isolated container topics. |
solr/cross-dc-manager/src/test/org/apache/solr/crossdc/manager/SolrAndKafkaIntegrationTest.java | Migrates primary integration tests. |
solr/cross-dc-manager/src/test/org/apache/solr/crossdc/manager/RetryQueueIntegrationTest.java | Migrates retry testing. |
solr/cross-dc-manager/src/test/org/apache/solr/crossdc/manager/KafkaContainerRule.java | Adds shared Kafka container lifecycle. |
solr/cross-dc-manager/src/test/org/apache/solr/crossdc/manager/DeleteByQueryToIdTest.java | Migrates delete-query testing. |
solr/cross-dc-manager/src/java/org/apache/solr/crossdc/manager/consumer/Util.java | Updates committed-offset API usage. |
solr/cross-dc-manager/gradle.lockfile | Refreshes manager dependencies. |
solr/cross-dc-manager/build.gradle | Adds Testcontainers and removes server dependencies. |
gradle/testing/randomization/policies/solr-tests.policy | Adds a global permission grant. |
gradle/libs.versions.toml | Declares Kafka 4 and Testcontainers Kafka. |
changelog/unreleased/SOLR-18300-update-kafka-v4.yml | Records the dependency update. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
| // Global override for local integration testing | ||
| grant { | ||
| permission java.security.AllPermission; | ||
| }; |
There was a problem hiding this comment.
I just pushed up a simpler test policy! Please @jaykay12 let me know that it works for you locally!
There was a problem hiding this comment.
hey @epugh , no this is the test that i am using to verify : SolrAndKafkaReindexTest
with dockerhub running in local, this test was passing earlier with global override. for the simpler policy(your changes) it is now faiiling, this is the error that i see.
does this test work for you locally with the simpler policy?
Uh oh!
There was an error while loading. Please reload this page.
| testImplementation libs.testcontainers.kafka | ||
Uh oh!
There was an error while loading. Please reload this page.
… into SOLR-18300-update-kafka-v4
| * <p>Skips the calling test (via {@link Assume}) instead of failing outright if | ||
| * Docker/Testcontainers isn't available in this environment. | ||
| */ | ||
| public class KafkaContainerRule extends ExternalResource { |
| // Docker/Testcontainers integration tests (KafkaContainerRule) do PATH scanning, | ||
| // spawn processes, and talk to the Docker daemon over a Unix socket -- none of | ||
| // which play well with a static SecurityManager policy. Same rationale as | ||
| // solr:modules:extraction, Solr's other Testcontainers consumer. | ||
| useSecurityManager = false |
There was a problem hiding this comment.
this is great! 💯
verified this working fine on local here: #4610 (comment)
epugh
commented
Aug 20, 2026
@jaykay12 i just pruned some more dependencies, can you confirm the tests DO run with Docker? |
jaykay12
left a comment
There was a problem hiding this comment.
precommit checks running fine locally. ✅
Uh oh!
There was an error while loading. Please reload this page.
Signed-off-by: jaykay12 <jalazkumar1208@gmail.com> Co-authored-by: SolrBot <solrbot@cominvent.com> Co-authored-by: Eric Pugh <epugh@opensourceconnections.com> (cherry picked from commit 8c750e2)
https://issues.apache.org/jira/browse/SOLR-18300
Description
This branch is checked-out from the branch which solrbot created.
This PR is a superset of #3274 (-> can be closed once this is merged)
There were few major & significant changes introduced in apache.kafka v4, which are being done in this PR alongside the normal version bumps.
Solution
We were using EmbeddedKafkaCluster which in V4 is no longer supported.
Went ahead with the TestContainer approach & updated all tests in cross-dc-manager module accordingly.
AI Usage Disclosure - Used Gemini Chat (3.5 Flash) - for the guidance & code snippets.
Tests
./gradlew check -x test-> ✅Checklist
Please review the following and check all that apply:
mainbranch../gradlew check.