Skip to content

[SPARK-30311][SS] Use KafkaConsumer.poll(Duration) instead of deprecated one - #26954

Closed
dongjoon-hyun wants to merge 2 commits into
apache:masterfrom
dongjoon-hyun:SPARK-30311
Closed

[SPARK-30311][SS] Use KafkaConsumer.poll(Duration) instead of deprecated one#26954
dongjoon-hyun wants to merge 2 commits into
apache:masterfrom
dongjoon-hyun:SPARK-30311

Conversation

@dongjoon-hyun

@dongjoon-hyundongjoon-hyun commented Dec 19, 2019

Copy link
Copy Markdown
Member

What changes were proposed in this pull request?

This PR replaces the deprecated KafkaConsumer.poll(long) with KafkaConsumer.poll(Duration). The old one is deprecated at Kafka 2.0.0.

Why are the changes needed?

This will use the latest API.

Does this PR introduce any user-facing change?

No.

How was this patch tested?

Pass the Jenkins.

dongjoon-hyun
dongjoon-hyun commented Dec 19, 2019
View reviewed changes
Comment threadexternal/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/KafkaOffsetReader.scala
package org.apache.spark.sql.kafka010

import java.{util => ju}
import java.time.{Duration => jDuration}

@dongjoon-hyundongjoon-hyunDec 19, 2019

Copy link
Copy Markdown
MemberAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This will avoid conflicts with import scala.concurrent.duration.Duration at line 27.

@HeartSaVioR

HeartSaVioR commented Dec 19, 2019

Copy link
Copy Markdown
Contributor

Sorry, but we tried it out and that cannot be simple replacement.

Please refer https://issues.apache.org/jira/browse/SPARK-28367 and #25135

We were waiting for Kafka 2.4.0 to deal with poll(0) replacement: Kafka 2.4.0 is now available, but if we have time to address, I'd like to give @gaborgsomogyi a chance to work for this, as he has been waiting for Kafka 2.4.0. He'll be back on next month.

@dongjoon-hyun

dongjoon-hyun commented Dec 19, 2019

Copy link
Copy Markdown
MemberAuthor

Oh, I didn't notice that effort. Let me take a look. Thanks, @HeartSaVioR .

@AmplabJenkins

AmplabJenkins commented Dec 19, 2019

Copy link
Copy Markdown

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/115585/
Test FAILed.

@dongjoon-hyun
dongjoon-hyun deleted the SPARK-30311 branch December 19, 2019 22:47
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

STRUCTURED STREAMING

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@dongjoon-hyun@HeartSaVioR@AmplabJenkins