[SPARK-2109] Setting SPARK_MEM for bin/pyspark does not work. by ScrapCodes · Pull Request #1050 · apache/spark

ScrapCodes · 2014-06-11T10:21:06Z

Trivial fix.

AmplabJenkins · 2014-06-11T10:22:05Z

Merged build triggered.

AmplabJenkins · 2014-06-11T10:22:11Z

Merged build started.

AmplabJenkins · 2014-06-11T11:05:44Z

Merged build finished. All automated tests passed.

AmplabJenkins · 2014-06-11T11:05:44Z

All automated tests passed.
Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/15676/

andrewor14 · 2014-06-11T19:19:42Z

LGTM

vanzin · 2014-06-11T21:36:16Z

This looks ok, but note that this isn't the only thing that can cause pyspark to not work.

e.g. "SPARK_PRINT_LAUNCH_COMMAND=1 bin/pyspark" also fails for the same reason. Perhaps a more long-term solution would be to not use stdout for communicating the port number...

AmplabJenkins · 2014-06-12T05:27:05Z

Merged build triggered.

AmplabJenkins · 2014-06-12T05:27:11Z

Merged build started.

ScrapCodes · 2014-06-12T05:27:30Z

@vanzin You are right, for now I have done the audit of all options available. Do you have an alternative? I did not look deeply enough. Will see if there is something simple can be done as an alternative.

AmplabJenkins · 2014-06-12T06:08:32Z

Merged build finished. All automated tests passed.

AmplabJenkins · 2014-06-12T06:08:32Z

All automated tests passed.
Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/15701/

vanzin · 2014-06-12T16:50:22Z

@ScrapCodes there are alternatives if you're willing to write more code. e.g., write the port to a separate file, or write the port with some specific formatting and look for it in the output instead of relying on the first line from the child process.

andrewor14 · 2014-06-12T17:18:57Z

Also, your latest fixes use the block EOF sometimes, and multiple lines of echo sometimes. It would be good if we could keep this consistent.

AmplabJenkins · 2014-06-13T06:57:05Z

Build triggered.

AmplabJenkins · 2014-06-13T06:57:12Z

Build started.

ScrapCodes · 2014-06-13T07:09:41Z

Hey @andrewor14, I sort of avoided EOF blocks for nested ifs, since that can look a bit less nicer because of no indentation.

AmplabJenkins · 2014-06-13T08:18:38Z

Build finished.

AmplabJenkins · 2014-06-13T08:18:38Z

Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/15754/

andrewor14 · 2014-06-13T20:02:52Z

Indentation is weird here

If I indent the contents of this EOF block, then those indentation will become part of the string that gets printed on the screen. I can still do it, if you think ? But I think some people will object to that.

andrewor14 · 2014-06-13T20:03:55Z

Hi @ScrapCodes. There are a couple of weird indentations, but other than that this looks good.

ScrapCodes · 2014-06-16T08:05:59Z

Hi @andrewor14, added a comment about them. Not sure if we can fix them.

vanzin · 2014-06-16T16:18:15Z

@ScrapCodes: in that case, wouldn't it be better to just use echo everywhere? Yeah, it's a little more verbose, but doesn't suffer from the indentation issue.

(It would be probably be cleaner to have an "error" function somewhere that does the correct redirection, but that would mean having a common script with utility functions that is sources from every script...)

andrewor14 · 2014-06-16T16:47:24Z

Yeah, I personally prefer echoes. The indentation thing is a little hard to track sometimes, especially because this is bash.

AmplabJenkins · 2014-06-17T07:04:47Z

Build triggered.

AmplabJenkins · 2014-06-17T07:04:54Z

Build started.

AmplabJenkins · 2014-06-17T08:17:27Z

Build finished.

AmplabJenkins · 2014-06-17T08:17:28Z

Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/15846/

ScrapCodes · 2014-06-17T08:47:45Z

Jenkins, retest this please.

AmplabJenkins · 2014-06-17T08:49:48Z

Build triggered.

AmplabJenkins · 2014-06-17T10:20:50Z

Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/15847/

pwendell · 2014-06-17T17:31:15Z

You are getting test failures because the patch doesn't merge cleanly.

AmplabJenkins · 2014-06-23T13:55:12Z

Merged build triggered.

AmplabJenkins · 2014-06-23T13:55:19Z

Merged build started.

AmplabJenkins · 2014-06-23T14:05:23Z

Merged build finished.

AmplabJenkins · 2014-06-23T14:05:23Z

Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/16026/

ScrapCodes · 2014-06-23T14:11:52Z

I just checked it merges cleanly and still these test failures.

ScrapCodes · 2014-06-23T14:12:06Z

Jenkins, retest this please.

AmplabJenkins · 2014-06-23T14:15:14Z

Merged build triggered.

AmplabJenkins · 2014-06-23T14:15:24Z

Merged build started.

AmplabJenkins · 2014-06-23T14:56:16Z

Merged build finished.

AmplabJenkins · 2014-06-23T14:56:16Z

Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/16028/

andrewor14 · 2014-06-23T17:56:19Z

Jenkins, test this please

AmplabJenkins · 2014-06-23T18:00:17Z

Merged build triggered.

AmplabJenkins · 2014-06-23T18:00:26Z

Merged build started.

AmplabJenkins · 2014-06-23T19:42:10Z

Merged build finished. All automated tests passed.

AmplabJenkins · 2014-06-23T19:42:10Z

All automated tests passed.
Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/16035/

andrewor14 · 2014-07-03T20:56:29Z

LGTM

Trivial fix. Author: Prashant Sharma <prashant.s@imaginea.com> Closes#1050 from ScrapCodes/SPARK-2109/pyspark-script-bug and squashes the following commits: 77072b9 [Prashant Sharma] Changed echos to redirect to STDERR. 13f48a0 [Prashant Sharma] [SPARK-2109] Setting SPARK_MEM for bin/pyspark does not work. (cherry picked from commit 731f683) Signed-off-by: Patrick Wendell <pwendell@gmail.com>

Trivial fix. Author: Prashant Sharma <prashant.s@imaginea.com> Closesapache#1050 from ScrapCodes/SPARK-2109/pyspark-script-bug and squashes the following commits: 77072b9 [Prashant Sharma] Changed echos to redirect to STDERR. 13f48a0 [Prashant Sharma] [SPARK-2109] Setting SPARK_MEM for bin/pyspark does not work.

andrewor14 reviewed Jun 13, 2014
View reviewed changes

ScrapCodes added 2 commits June 23, 2014 19:19

[SPARK-2109] Setting SPARK_MEM for bin/pyspark does not work.
13f48a0

Changed echos to redirect to STDERR.
77072b9

andrewor14 mentioned this pull request Jun 24, 2014
[SPARK-2242] HOTFIX: pyspark shell hangs on simple job #1178
Closed

asfgit closed this in 731f683Jul 3, 2014

ScrapCodes deleted the SPARK-2109/pyspark-script-bug branch June 3, 2015 06:01

Uh oh!

Conversation

ScrapCodes commented Jun 11, 2014

Uh oh!

AmplabJenkins commented Jun 11, 2014

Uh oh!

AmplabJenkins commented Jun 11, 2014

Uh oh!

AmplabJenkins commented Jun 11, 2014

Uh oh!

AmplabJenkins commented Jun 11, 2014

Uh oh!

andrewor14 commented Jun 11, 2014

Uh oh!

vanzin commented Jun 11, 2014

Uh oh!

AmplabJenkins commented Jun 12, 2014

Uh oh!

AmplabJenkins commented Jun 12, 2014

Uh oh!

ScrapCodes commented Jun 12, 2014

Uh oh!

AmplabJenkins commented Jun 12, 2014

Uh oh!

AmplabJenkins commented Jun 12, 2014

Uh oh!

vanzin commented Jun 12, 2014

Uh oh!

andrewor14 commented Jun 12, 2014

Uh oh!

AmplabJenkins commented Jun 13, 2014

Uh oh!

AmplabJenkins commented Jun 13, 2014

Uh oh!

ScrapCodes commented Jun 13, 2014

Uh oh!

AmplabJenkins commented Jun 13, 2014

Uh oh!

AmplabJenkins commented Jun 13, 2014

Uh oh!

andrewor14Jun 13, 2014

Choose a reason for hiding this comment

Uh oh!

ScrapCodesJun 16, 2014

Choose a reason for hiding this comment

Uh oh!

andrewor14 commented Jun 13, 2014

Uh oh!

ScrapCodes commented Jun 16, 2014

Uh oh!

vanzin commented Jun 16, 2014

Uh oh!

andrewor14 commented Jun 16, 2014

Uh oh!

AmplabJenkins commented Jun 17, 2014

Uh oh!

AmplabJenkins commented Jun 17, 2014

Uh oh!

AmplabJenkins commented Jun 17, 2014

Uh oh!

AmplabJenkins commented Jun 17, 2014

Uh oh!

ScrapCodes commented Jun 17, 2014

Uh oh!

AmplabJenkins commented Jun 17, 2014

Uh oh!

AmplabJenkins commented Jun 17, 2014

Uh oh!

pwendell commented Jun 17, 2014

Uh oh!

AmplabJenkins commented Jun 23, 2014

Uh oh!

AmplabJenkins commented Jun 23, 2014

Uh oh!

AmplabJenkins commented Jun 23, 2014

Uh oh!

AmplabJenkins commented Jun 23, 2014

Uh oh!

ScrapCodes commented Jun 23, 2014

Uh oh!