[SPARK-6027][SPARK-5546] Fixed --jar and --packages not working for KafkaUtils and improved error message by tdas · Pull Request #4779 · apache/spark

tdas · 2015-02-26T04:22:10Z

The problem with SPARK-6027 in short is that JARs like the kafka-assembly.jar does not work in python as the added JAR is not visible in the classloader used by Py4J. Py4J uses Class.forName(), which does not uses the systemclassloader, but the JARs are only visible in the Thread's contextclassloader. So this back uses the context class loader to create the KafkaUtils dstream object. This works for both cases where the Kafka libraries are added with --jars spark-streaming-kafka-assembly.jar or with --packages spark-streaming-kafka

Also improves the error message.

@davies

SparkQA · 2015-02-26T04:27:35Z

Test build #27980 has started for PR 4779 at commit 7b88be8.

This patch merges cleanly.

SparkQA · 2015-02-26T04:28:32Z

Test build #27980 has finished for PR 4779 at commit 7b88be8.

This patch fails Python style tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- class KafkaUtilsPythonHelper

AmplabJenkins · 2015-02-26T04:28:33Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/27980/
Test FAILed.

jerryshao · 2015-02-26T05:02:34Z

Hi @tdas , a stupid question, why previous way of --driver-class-path is not good, since if we use --jars, we have to create every helper function for Python, I'm a little confused.

davies · 2015-02-26T05:46:31Z

remove java_import()

tdas · 2015-02-26T05:52:12Z

@jerryshao the previous way required --driver-class-path as well as --jars (or --driver-class-path and --executor-class-path). The existing message was incorrect, @davies and I missed it in the original PR because we probably never tested in distributed mode. Instead, --jars is supposed to be a single solution for adding any jar to both driver and executor. It works for Scala/Java but does not work correctly for Python, hence this workaround.

SparkQA · 2015-02-26T05:52:38Z

Test build #27984 has started for PR 4779 at commit c1fdf35.

This patch merges cleanly.

davies · 2015-02-26T05:59:01Z

LGTM, the helper class actually simplify things.

SparkQA · 2015-02-26T06:03:00Z

Test build #27987 has started for PR 4779 at commit fb16b04.

This patch merges cleanly.

SparkQA · 2015-02-26T07:15:59Z

Test build #27984 has finished for PR 4779 at commit c1fdf35.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- class KafkaUtilsPythonHelper

AmplabJenkins · 2015-02-26T07:16:03Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/27984/
Test PASSed.

SparkQA · 2015-02-26T07:22:23Z

Test build #27987 has finished for PR 4779 at commit fb16b04.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- class KafkaUtilsPythonHelper

AmplabJenkins · 2015-02-26T07:22:27Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/27987/
Test PASSed.

andrewor14 · 2015-02-26T17:46:23Z

does it still work if this is strictly private? Just wondering

I am not sure if I will be able to call from python if this is strictly private. I havent tested it though. I dont think that;s a big deal, as long as its hidden from public view

andrewor14 · 2015-02-26T21:45:57Z

LGTM as discussed with @tdas offline we're gonna change it to private. This is going into master and 1.3.

…afkaUtils and improved error message The problem with SPARK-6027 in short is that JARs like the kafka-assembly.jar does not work in python as the added JAR is not visible in the classloader used by Py4J. Py4J uses Class.forName(), which does not uses the systemclassloader, but the JARs are only visible in the Thread's contextclassloader. So this back uses the context class loader to create the KafkaUtils dstream object. This works for both cases where the Kafka libraries are added with --jars spark-streaming-kafka-assembly.jar or with --packages spark-streaming-kafka Also improves the error message. davies Author: Tathagata Das <tathagata.das1565@gmail.com> Closes#4779 from tdas/kafka-python-fix and squashes the following commits: fb16b04 [Tathagata Das] Removed import c1fdf35 [Tathagata Das] Fixed long line and improved documentation 7b88be8 [Tathagata Das] Fixed --jar not working for KafkaUtils and improved error message (cherry picked from commit aa63f63) Signed-off-by: Andrew Or <andrew@databricks.com>

…oading issue This patch upgrades Py4J from 0.9.1 to 0.9.2 in order to include a patch which modifies Py4J to use the current thread's ContextClassLoader when performing reflection / class loading. This is necessary in order to fix [SPARK-5185](https://issues.apache.org/jira/browse/SPARK-5185), a longstanding issue affecting the use of `--jars` and `--packages` in PySpark. In order to demonstrate that the fix works, I removed the workarounds which were added as part of [SPARK-6027](https://issues.apache.org/jira/browse/SPARK-6027) / #4779 and other patches. Py4J diff: py4j/py4j@0.9.1...0.9.2 /cc zsxwing tdas davies brkyvz Author: Josh Rosen <joshrosen@databricks.com> Closes#11687 from JoshRosen/py4j-0.9.2.

…oading issue This patch upgrades Py4J from 0.9.1 to 0.9.2 in order to include a patch which modifies Py4J to use the current thread's ContextClassLoader when performing reflection / class loading. This is necessary in order to fix [SPARK-5185](https://issues.apache.org/jira/browse/SPARK-5185), a longstanding issue affecting the use of `--jars` and `--packages` in PySpark. In order to demonstrate that the fix works, I removed the workarounds which were added as part of [SPARK-6027](https://issues.apache.org/jira/browse/SPARK-6027) / apache#4779 and other patches. Py4J diff: py4j/py4j@0.9.1...0.9.2 /cc zsxwing tdas davies brkyvz Author: Josh Rosen <joshrosen@databricks.com> Closesapache#11687 from JoshRosen/py4j-0.9.2.

Fixed --jar not working for KafkaUtils and improved error message
7b88be8

davies reviewed Feb 26, 2015
View reviewed changes

Comment threadpython/pyspark/streaming/kafka.py
daviesFeb 26, 2015
Copy link
Copy Markdown
Contributor
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
remove java_import()

Fixed long line and improved documentation
c1fdf35

Removed import
fb16b04

andrewor14 reviewed Feb 26, 2015
View reviewed changes

tdas changed the title ~~[SPARK-6027][SPARK-5546] Fixed --jar not working for KafkaUtils and improved error message~~[SPARK-6027][SPARK-5546] Fixed --jar and --packages not working for KafkaUtils and improved error messageFeb 26, 2015

asfgit closed this in aa63f63Feb 26, 2015

JoshRosen mentioned this pull request Mar 13, 2016
[SPARK-13848][SPARK-5185] Update to Py4J 0.9.2 in order to fix classloading issue #11687
Closed

Uh oh!

Conversation

tdas commented Feb 26, 2015

Uh oh!

SparkQA commented Feb 26, 2015

Uh oh!

SparkQA commented Feb 26, 2015

Uh oh!

AmplabJenkins commented Feb 26, 2015

Uh oh!

jerryshao commented Feb 26, 2015

Uh oh!

daviesFeb 26, 2015

Choose a reason for hiding this comment

Uh oh!

tdas commented Feb 26, 2015

Uh oh!

SparkQA commented Feb 26, 2015

Uh oh!

davies commented Feb 26, 2015

Uh oh!

SparkQA commented Feb 26, 2015

Uh oh!

SparkQA commented Feb 26, 2015

Uh oh!

AmplabJenkins commented Feb 26, 2015

Uh oh!

SparkQA commented Feb 26, 2015

Uh oh!

AmplabJenkins commented Feb 26, 2015

Uh oh!

andrewor14Feb 26, 2015

Choose a reason for hiding this comment

Uh oh!

tdasFeb 26, 2015

Choose a reason for hiding this comment

Uh oh!

andrewor14 commented Feb 26, 2015

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants