[SPARK-6211][Streaming] Add Python Kafka API unit test by jerryshao · Pull Request #4961 · apache/spark

jerryshao · 2015-03-10T09:11:18Z

Refactor the Kafka unit test and add Python API support. CC @tdas @davies please help to review, thanks a lot.

SparkQA · 2015-03-10T09:12:48Z

Test build #28427 has started for PR 4961 at commit d4f2e5b.

This patch merges cleanly.

SparkQA · 2015-03-10T10:18:59Z

Test build #28427 has finished for PR 4961 at commit d4f2e5b.

This patch fails PySpark unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- class EmbeddedZookeeper(val zkConnect: String)

AmplabJenkins · 2015-03-10T10:19:03Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/28427/
Test FAILed.

SparkQA · 2015-03-10T12:52:45Z

Test build #28429 has started for PR 4961 at commit 4898694.

This patch merges cleanly.

SparkQA · 2015-03-10T14:00:46Z

Test build #28429 has finished for PR 4961 at commit 4898694.

This patch fails PySpark unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- class EmbeddedZookeeper(val zkConnect: String)

AmplabJenkins · 2015-03-10T14:00:50Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/28429/
Test FAILed.

jerryshao · 2015-03-10T14:14:01Z

Jenkins, retest this please.

SparkQA · 2015-03-10T14:18:06Z

Test build #28431 has started for PR 4961 at commit 4898694.

This patch merges cleanly.

SparkQA · 2015-03-10T15:27:14Z

Test build #28431 has finished for PR 4961 at commit 4898694.

This patch fails PySpark unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- class EmbeddedZookeeper(val zkConnect: String)

AmplabJenkins · 2015-03-10T15:27:18Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/28431/
Test FAILed.

SparkQA · 2015-03-11T01:58:07Z

Test build #28458 has started for PR 4961 at commit f66a067.

This patch merges cleanly.

SparkQA · 2015-03-11T03:07:18Z

Test build #28458 has finished for PR 4961 at commit f66a067.

This patch fails PySpark unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- class EmbeddedZookeeper(val zkConnect: String)

AmplabJenkins · 2015-03-11T03:07:22Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/28458/
Test FAILed.

jerryshao · 2015-03-11T07:03:09Z

Looks like maven command in run-tests script do not generate kafka-assembly fat jar, only a single module empty jar. I took a look at the pom file under kafka-assembly, actually has shade plugin and enabled under package, I'm not sure why mvn package from parent module cannot generate kafka-assembly fat jar, only one assembly jar in module assembly is generated, is it a bug or just the right behavior?

Currently what I did is to use mvn -am -pl "external/kafka-assembly" package to generate a fat jar, do you have any suggestions, @tdas. Thanks a lot.

tdas · 2015-03-12T18:22:41Z

This seems like a bug that was fixed couple of weeks ago. Are you sure you have merged with master recently. In fact the PR has merge conflicts.

tdas · 2015-03-12T18:30:48Z

Can you add one line to say why this is in /main and not /test.

tdas · 2015-03-12T22:21:55Z

Other than the assembly issue, its looks quite good. Only a few minor comments.

jerryshao · 2015-03-13T01:32:13Z

Thanks a lot @tdas for your review, actually I rebased the code in my local environment and tried to test again, seems a recent merged PR #3916 changed the behavior of pyspark, so previously --jars argument that can be worked in unit test are failed to work now, I'm trying to figure out a way to enable this in unit test.

jerryshao · 2015-04-10T02:50:16Z

Seems recent merged PR #5261 changed the script and make my test fail, I will fix it.

SparkQA · 2015-04-10T03:08:22Z

Test build #29999 has started for PR 4961 at commit ee4b919.

SparkQA · 2015-04-10T04:36:02Z

Test build #29999 has finished for PR 4961 at commit ee4b919.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- class FlattenedValuesSerializer(BatchedSerializer):
- class ExternalList(object):
- class ExternalListOfList(ExternalList):
- class GroupByKey(object):
- class ExternalGroupBy(ExternalMerger):
This patch does not change any dependencies.

AmplabJenkins · 2015-04-10T04:36:07Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/29999/
Test PASSed.

jerryshao · 2015-04-10T04:38:34Z

Jenkins, retest this please.

SparkQA · 2015-04-10T04:43:30Z

Test build #30006 has started for PR 4961 at commit ee4b919.

tdas · 2015-04-10T04:48:20Z

Jenkins, retest this please.
On Apr 9, 2015 6:52 PM, "Apache Spark QA" notifications@github.com wrote:

Test build #29991 has finished
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/29991/consoleFull
for PR 4961 at commit 1cda32a
1cda32a
.
This patch fails PySpark unit tests.
This patch merges cleanly.
This patch adds no public classes.
This patch does not change any dependencies.
—
Reply to this email directly or view it on GitHub
#4961 (comment).

tdas · 2015-04-10T05:35:23Z

LGTM. Will merge if it passes.

SparkQA · 2015-04-10T06:08:25Z

Test build #30006 has finished for PR 4961 at commit ee4b919.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.
This patch does not change any dependencies.

AmplabJenkins · 2015-04-10T06:08:29Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/30006/
Test PASSed.

tdas · 2015-04-10T06:12:02Z

Merging this.

tdas · 2015-04-10T06:15:59Z

Thanks @jerryshao for doing this!

tdas reviewed Mar 12, 2015
View reviewed changes

jerryshao force-pushed the SPARK-6211 branch from f66a067 to d290955Compare March 13, 2015 05:27

jerryshao added 13 commits April 10, 2015 10:00

Code style changes
fde1213

Fix bug in Jenkins test
8102d6e

Add more debug info in Shell
6020b00

Add kafka-assembly in run-tests
8ad442f

Fix rebase bugs
64d9877

Fix bugs and address the issues
61a04f0

Still fix the bug
0f1b7ce

Address the issues
8a2f3e2

Update the code according
f889657

Style fix
40b47a3

Fix rebase issue
0708bb1

Address the commits
92912d1

Address the comments
82c756e

jerryshao force-pushed the SPARK-6211 branch from 1cda32a to ee4b919Compare April 10, 2015 03:05

Fixed newly merged issue
ee4b919

asfgit closed this in 3290d2dApr 10, 2015

jerryshao mentioned this pull request Apr 23, 2015
[SPARK-7050][build] Fix Python Kafka test assembly jar not found issue under Maven build #5632
Closed

Uh oh!

Conversation

jerryshao commented Mar 10, 2015

Uh oh!

SparkQA commented Mar 10, 2015

Uh oh!

SparkQA commented Mar 10, 2015

Uh oh!

AmplabJenkins commented Mar 10, 2015

Uh oh!

SparkQA commented Mar 10, 2015

Uh oh!

SparkQA commented Mar 10, 2015

Uh oh!

AmplabJenkins commented Mar 10, 2015

Uh oh!

jerryshao commented Mar 10, 2015

Uh oh!

SparkQA commented Mar 10, 2015

Uh oh!

SparkQA commented Mar 10, 2015

Uh oh!

AmplabJenkins commented Mar 10, 2015

Uh oh!

SparkQA commented Mar 11, 2015

Uh oh!

SparkQA commented Mar 11, 2015

Uh oh!

AmplabJenkins commented Mar 11, 2015

Uh oh!

jerryshao commented Mar 11, 2015

Uh oh!

tdas commented Mar 12, 2015

Uh oh!

tdasMar 12, 2015

Choose a reason for hiding this comment

Uh oh!

tdas commented Mar 12, 2015

Uh oh!

jerryshao commented Mar 13, 2015

Uh oh!

jerryshao commented Apr 10, 2015

Uh oh!

SparkQA commented Apr 10, 2015

Uh oh!

SparkQA commented Apr 10, 2015

Uh oh!

AmplabJenkins commented Apr 10, 2015

Uh oh!

jerryshao commented Apr 10, 2015

Uh oh!

SparkQA commented Apr 10, 2015

Uh oh!

tdas commented Apr 10, 2015

Uh oh!

tdas commented Apr 10, 2015

Uh oh!

SparkQA commented Apr 10, 2015

Uh oh!

AmplabJenkins commented Apr 10, 2015

Uh oh!

tdas commented Apr 10, 2015

Uh oh!

tdas commented Apr 10, 2015

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants