Uh oh!
There was an error while loading. Please reload this page.
[SPARK-1466] Raise exception if pyspark Gateway process doesn't start. - #383
[SPARK-1466] Raise exception if pyspark Gateway process doesn't start.#383kayousterhout wants to merge 1 commit into
Conversation
kayousterhout
commented
Apr 10, 2014
This should be backported to 0.9 and 1.0 |
kayousterhout
commented
Apr 10, 2014
BTW this is a much bigger issue with iPython notebook -- if you're running in the console, you get the wrong error (with parsing the int) but also the correct error. If you're running in iPython notebook, you only get the wrong error, making this very annoying to debug. |
AmplabJenkins
commented
Apr 10, 2014
Merged build triggered. |
AmplabJenkins
commented
Apr 10, 2014
Merged build started. |
AmplabJenkins
commented
Apr 10, 2014
Merged build finished. |
AmplabJenkins
commented
Apr 10, 2014
Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/14017/ |
pwendell
commented
Apr 12, 2014
Jenkins, retest this please. |
AmplabJenkins
commented
Apr 12, 2014
Merged build triggered. |
AmplabJenkins
commented
Apr 12, 2014
Merged build started. |
AmplabJenkins
commented
Apr 12, 2014
Merged build finished. |
AmplabJenkins
commented
Apr 12, 2014
Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/14069/ |
There was a problem hiding this comment.
Maybe this should say "Launching GatewayServer failed"? It will be more informative, otherwise people will think something is wrong with SparkContext itself.
mateiz
commented
Apr 18, 2014
@kayousterhout not sure if you saw my comment, this looks good but the exception message is somewhat confusing. It would be good to update that. |
pwendell
commented
Apr 18, 2014
Jenkins, retest this please. |
AmplabJenkins
commented
Apr 18, 2014
Merged build triggered. |
AmplabJenkins
commented
Apr 18, 2014
Merged build started. |
kayousterhout
commented
Apr 18, 2014
I did but haven't had time to figure out why the tests are failing (the tests don't run properly on my laptop). Hoping this was a Jenkins issue and the re-launched tests pass. |
AmplabJenkins
commented
Apr 18, 2014
Merged build finished. |
AmplabJenkins
commented
Apr 18, 2014
Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/14230/ |
Also include stderr output to help user debug startup issue.
AmplabJenkins
commented
Jun 10, 2014
Merged build triggered. |
AmplabJenkins
commented
Jun 10, 2014
Merged build started. |
AmplabJenkins
commented
Jun 10, 2014
Merged build finished. All automated tests passed. |
AmplabJenkins
commented
Jun 10, 2014
All automated tests passed. |
mateiz
commented
Jun 10, 2014
It looks like the tests magically passed now! Is this good to go? |
kayousterhout
commented
Jun 10, 2014
Yup! I just rebased so it should merge cleanly on master. Sent from my iPhone On Jun 9, 2014, at 7:03 PM, Matei Zaharia notifications@github.com wrote:
|
If the gateway process fails to start correctly (e.g., because JAVA_HOME isn't set correctly, there's no Spark jar, etc.), right now pyspark fails because of a very difficult-to-understand error, where we try to parse stdout to get the port where Spark started and there's nothing there. This commit properly catches the error and throws an exception that includes the stderr output for much easier debugging. Thanks to @shivaram and @stogers for helping to fix this issue! Author: Kay Ousterhout <kayousterhout@gmail.com> Closesapache#383 from kayousterhout/pyspark and squashes the following commits: 36dd54b [Kay Ousterhout] [SPARK-1466] Raise exception if Gateway process doesn't start.
If the gateway process fails to start correctly (e.g., because JAVA_HOME isn't set correctly, there's no Spark jar, etc.), right now pyspark fails because of a very difficult-to-understand error, where we try to parse stdout to get the port where Spark started and there's nothing there. This commit properly catches the error and throws an exception that includes the stderr output for much easier debugging. Thanks to @shivaram and @stogers for helping to fix this issue! Author: Kay Ousterhout <kayousterhout@gmail.com> Closesapache#383 from kayousterhout/pyspark and squashes the following commits: 36dd54b [Kay Ousterhout] [SPARK-1466] Raise exception if Gateway process doesn't start.
…he#383) This makes executors consistent with the driver. Note that SPARK_EXTRA_CLASSPATH isn't set anywhere by Spark itself, but it's primarily meant to be set by images that inherit from the base driver/executor images.
…he#383) This makes executors consistent with the driver. Note that SPARK_EXTRA_CLASSPATH isn't set anywhere by Spark itself, but it's primarily meant to be set by images that inherit from the base driver/executor images.
Apply patches for SPARK-24531 to fix tests
Diable S3 test cases in fusioncloud job
If the gateway process fails to start correctly (e.g., because JAVA_HOME isn't set correctly, there's no Spark jar, etc.), right now pyspark fails because of a very difficult-to-understand error, where we try to parse stdout to get the port where Spark started and there's nothing there. This commit properly catches the error and throws an exception that includes the stderr output for much easier debugging.
Thanks to @shivaram and @stogers for helping to fix this issue!