[SPARK-12177] [STREAMING] Update KafkaDStreams to new Kafka 0.9 Consu…

[nikit-os]

…mer API

[AmplabJenkins]

Can one of the admins verify this patch?

[holdenk]

This comment style is inconsistent with the rest of Spark (also makes it difficult to see what parts of the comment has/hasn't changed).

[jerryshao]

There's a lot of duplications copied from the original code, is it possible to consolidate it? Also if current Kafka client is wire-compatible with Kafka 0.9 brokers, we could still use the current client and wait until most of the uses upgrade the Kafka cluster to 0.9 to change the client. From my thinking maintaining two versions of Kafka in Spark will increase the maintenance burden.

[nikit-os]

I did this duplicates because I didn't want to broke the current implementation. Do you have some ideas how to do consolidation between this two versions? Maybe we should use something like shim classes for this - set of abstract classes and 2 submodules with old and new implementations?
I can not make such decisions, but in my opinion we may leave current 0.8 implementation as legacy and evolve new 0.9 implementation.

[praveend]

probably writing 0L would be better rather than (long) casting

[praveend]

Hi @nikit-os

I have made a note of some minor observations and suggestions on some of the test cases related to using new consumer. Have a look and let me know if you will need any further info from me on them.

Thanks

Praveen

[nikit-os]

Hi @praveend !

Thank you for review! I have corrected mistakes.