[SPARK-17782][STREAMING][BUILD] Add Kafka 0.10 project to build modules

[hvanhovell]

What changes were proposed in this pull request?

This PR adds the Kafka 0.10 subproject to the build infrastructure. This makes sure Kafka 0.10 tests are only triggers when it or of its dependencies change.

[hvanhovell]

cc @koeninger any idea why this flaky?

[koeninger]

I have generally been unable to reproduce these kinds of test failures on my local environment, and don't have access to the build server, so trying fix without repro is pretty much shooting randomly in the dark. It does seem unfortunate to me that we're effectively doing full integration tests on every PR, even if a patch has changed something (e.g. MLLib) that couldn't possibly affect the modules in /external

[SparkQA]

Test build #66361 has finished for PR 15355 at commit b7074d4.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[rxin]

Actually we do have the infra to not run these tests if it is just an unrelated module change. Was those not setup for Kafka?

[srowen]

Good point, I don't see separate config for the 0.10 module in dev/sparktestsupport/modules.py though there is for 0.8 module. That could be a better way forward.

[SparkQA]

Test build #66461 has finished for PR 15355 at commit 0bac9ff.

• This patch fails executing the dev/run-tests script.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #66463 has finished for PR 15355 at commit 5ba78b6.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[zsxwing]

@koeninger you can download the unit test logs from https://amplab.cs.berkeley.edu/jenkins/job/spark-master-test-sbt-hadoop-2.6/1756/artifact/

I saw the offsets of the first batch (pat1 0 16 16 should be pat1 0 0 16) was wrong in the unit test logs:

16/10/05 20:42:46.313 streaming-start INFO JobGenerator: Started JobGenerator at 1475725367000 ms
16/10/05 20:42:46.313 streaming-start INFO JobScheduler: Started JobScheduler
16/10/05 20:42:46.313 pool-1-thread-1-ScalaTest-running-DirectKafkaStreamSuite INFO StreamingContext: StreamingContext started
16/10/05 20:42:47.015 JobGenerator INFO JobScheduler: Added jobs for time 1475725367000 ms
16/10/05 20:42:47.016 JobScheduler INFO JobScheduler: Starting job streaming job 1475725367000 ms.0 from job set of time 1475725367000 ms
16/10/05 20:42:47.017 streaming-job-executor-0 INFO DirectKafkaStreamSuite: pat2 0 3 16
16/10/05 20:42:47.017 streaming-job-executor-0 INFO DirectKafkaStreamSuite: pat1 0 16 16

Seems the earliest config does not work.

[koeninger]

@zsxwing good eye, thanks. It's not that auto.offset.reset.earliest doesn't work, it's that there's a potential race condition that poll gets called twice slowly enough for consumer position to be modified before topicpartitions are paused.

#15387

should address that.

It's something that whoever works on the duplicated equivalent code in the structured streaming module is going to have to address, also.

[hvanhovell]

I have re-enables the kafka test. This PR now only contains a change to build infrastructure.

[srowen]

This looks good and will merge as soon as it passes, though the actual fix will be in the other PR.

[SparkQA]

Test build #66487 has finished for PR 15355 at commit a2ba414.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[srowen]

Merged to master

[zsxwing]

Also check-picked this one into branch 2.0 since it's also helpful for 2.0 backport PRs.