Uh oh!
There was an error while loading. Please reload this page.
[SPARK-30336][SQL][SS] Move Kafka consumer-related classes to its own package - #26991
[SPARK-30336][SQL][SS] Move Kafka consumer-related classes to its own package#26991HeartSaVioR wants to merge 3 commits into
Conversation
| import java.{util => ju} | ||
| import scala.collection.mutable.ArrayBuffer |
There was a problem hiding this comment.
I've also cleaned up unused imports and remove unnecessary line (two empty lines between the last import statement and class definition)
| @@ -218,4 +218,3 @@ private[kafka010] object InternalKafkaConsumerPool { | |||
| } | |||
| } | |||
| } | |||
There was a problem hiding this comment.
There were two empty lines which can be just one empty line.
SparkQA
commented
Dec 24, 2019
Test build #115666 has finished for PR 26991 at commit
|
SparkQA
commented
Dec 24, 2019
Test build #115667 has finished for PR 26991 at commit
|
srowen
left a comment
There was a problem hiding this comment.
Looks OK as a cleanup, pending tests
HeartSaVioR
commented
Dec 24, 2019
retest this, please |
SparkQA
commented
Dec 24, 2019
Test build #115751 has finished for PR 26991 at commit
|
SparkQA
commented
Dec 26, 2019
Test build #115793 has finished for PR 26991 at commit
|
HeartSaVioR
commented
Dec 30, 2019
cc. @srowen Gently ping. |
srowen
left a comment
There was a problem hiding this comment.
If tests pass and it's just a refactoring, OK.
I guess you also made everything private to .consumer that you can?
HeartSaVioR
commented
Dec 30, 2019
Yes, the KafkaDataConsumer itself, and acquire and release are open to other package (kafka010). Others are private to consumer. |
srowen
commented
Dec 31, 2019
Merged to master |
HeartSaVioR
commented
Dec 31, 2019
Thamks for reviewing and merging! Happy new year! |
What changes were proposed in this pull request?
There're too many classes placed in a single package "org.apache.spark.sql.kafka010" which classes can be grouped by purpose.
As a part of change in SPARK-21869 (#26845), we moved out producer related classes to "org.apache.spark.sql.kafka010.producer" and only expose necessary classes/methods to the outside of package. This patch applies the same to consumer related classes.
Why are the changes needed?
Described above.
Does this PR introduce any user-facing change?
No.
How was this patch tested?
Existing UTs.