Skip to content

[SPARK-6120] [mllib] Warnings about memory in tree, ensemble model save - #4864

Closed
jkbradley wants to merge 2 commits into
apache:masterfrom
jkbradley:dt-save-heap
Closed

[SPARK-6120] [mllib] Warnings about memory in tree, ensemble model save#4864
jkbradley wants to merge 2 commits into
apache:masterfrom
jkbradley:dt-save-heap

Conversation

@jkbradley

@jkbradleyjkbradley commented Mar 3, 2015

Copy link
Copy Markdown
Member

Issue: When the Python DecisionTree example in the programming guide is run, it runs out of Java Heap Space when using the default memory settings for the spark shell.

This prints a warning.

CC: @mengxr

@SparkQA

SparkQA commented Mar 3, 2015

Copy link
Copy Markdown

Test build #28211 has started for PR 4864 at commit 7ecb1ed.

  • This patch merges cleanly.

mengxr
mengxr reviewed Mar 3, 2015
View reviewed changes
Comment threadmllib/src/main/scala/org/apache/spark/mllib/tree/model/DecisionTreeModel.scala Outdated

@mengxrmengxrMar 3, 2015

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Move this into the the if (sc.isLocal) branch.

@SparkQA

SparkQA commented Mar 3, 2015

Copy link
Copy Markdown

Test build #28213 has started for PR 4864 at commit 02e8daf.

  • This patch merges cleanly.

@SparkQA

SparkQA commented Mar 3, 2015

Copy link
Copy Markdown

Test build #28213 has finished for PR 4864 at commit 02e8daf.

  • This patch fails Spark unit tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@AmplabJenkins

AmplabJenkins commented Mar 3, 2015

Copy link
Copy Markdown

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/28213/
Test FAILed.

@SparkQA

SparkQA commented Mar 3, 2015

Copy link
Copy Markdown

Test build #28211 has finished for PR 4864 at commit 7ecb1ed.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@AmplabJenkins

AmplabJenkins commented Mar 3, 2015

Copy link
Copy Markdown

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/28211/
Test PASSed.

asfgit pushed a commit that referenced this pull request Mar 3, 2015
@jkbradley@mengxr
[SPARK-6120] [mllib] Warnings about memory in tree, ensemble model save
81648a7
Issue: When the Python DecisionTree example in the programming guide is run, it runs out of Java Heap Space when using the default memory settings for the spark shell.
This prints a warning.
CC: mengxr
Author: Joseph K. Bradley <joseph@databricks.com>
Closes#4864 from jkbradley/dt-save-heap and squashes the following commits:
02e8daf [Joseph K. Bradley] fixed based on code review
7ecb1ed [Joseph K. Bradley] Added warnings about memory when calling tree and ensemble model save with too small a Java heap size
(cherry picked from commit c2fe3a6)
Signed-off-by: Xiangrui Meng <meng@databricks.com>
@mengxr

mengxr commented Mar 3, 2015

Copy link
Copy Markdown
Contributor

LGTM. Merged into master and branch-1.3. Thanks!

@asfgitasfgit closed this in c2fe3a6Mar 3, 2015
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@jkbradley@SparkQA@AmplabJenkins@mengxr