Uh oh!
There was an error while loading. Please reload this page.
[SPARK-19064][PySpark]Fix pip installing of sub components - #16465
Closed
holdenk wants to merge 2 commits into
Closed
[SPARK-19064][PySpark]Fix pip installing of sub components#16465holdenk wants to merge 2 commits into
holdenk wants to merge 2 commits into
Conversation
…stribution scripts to be more explicit about cleanup. Also add pypandoc to dev requirements file since we want it for publishing
SparkQA
commented
Jan 4, 2017
Test build #70848 has finished for PR 16465 at commit
|
holdenk
commented
Jan 5, 2017
ContributorAuthor
cc @JoshRosen who reviewed the original PR and is probably the most familiar. |
viirya
reviewed
Jan 5, 2017
| from pyspark.sql import SparkSession | ||
| from pyspark.ml.param import Params | ||
| from pyspark.mllib.linalg import * |
Member
There was a problem hiding this comment.
Is it better to import pyspark.ml.linalg or both?
ContributorAuthor
There was a problem hiding this comment.
This just checks one sub component from each, we could import each with rename I suppose but not sure it would do much?
holdenk
commented
Jan 10, 2017
ContributorAuthor
Gentle ping for @JoshRosen to @davies maybe? |
holdenk
commented
Jan 24, 2017
ContributorAuthor
Small follow up ping for @JoshRosen to @davies maybe? |
srowen
approved these changes
Jan 24, 2017
srowen
left a comment
Member
There was a problem hiding this comment.
I'd say you can merge it yourself !
holdenk
commented
Jan 25, 2017
ContributorAuthor
Sounds like a plan, I think should should probably be on the 2.1 branch as well so I'll go bug someone who has done backports to make sure I do that part right :) |
felixcheung
commented
Jan 25, 2017
via email
Member
Holden, https://github.com/apache/spark/blob/master/dev/merge_spark_pr.py works well for merging to master and back port to any branch :) unless there is conflict then it would be easier with a separate PR.
Have fun! |
asfgit pushed a commit
that referenced
this pull request
Jan 25, 2017
## What changes were proposed in this pull request? Fix instalation of mllib and ml sub components, and more eagerly cleanup cache files during test script & make-distribution. ## How was this patch tested? Updated sanity test script to import mllib and ml sub-components. Author: Holden Karau <holden@us.ibm.com> Closes#16465 from holdenk/SPARK-19064-fix-pip-install-sub-components. (cherry picked from commit 965c82d) Signed-off-by: Holden Karau <holden@us.ibm.com>
uzadude pushed a commit
to uzadude/spark
that referenced
this pull request
Jan 27, 2017
## What changes were proposed in this pull request? Fix instalation of mllib and ml sub components, and more eagerly cleanup cache files during test script & make-distribution. ## How was this patch tested? Updated sanity test script to import mllib and ml sub-components. Author: Holden Karau <holden@us.ibm.com> Closesapache#16465 from holdenk/SPARK-19064-fix-pip-install-sub-components.
holdenk
commented
Feb 2, 2017
ContributorAuthor
Merged into master & branch-2.1 |
cmonkey pushed a commit
to cmonkey/spark
that referenced
this pull request
Feb 15, 2017
## What changes were proposed in this pull request? Fix instalation of mllib and ml sub components, and more eagerly cleanup cache files during test script & make-distribution. ## How was this patch tested? Updated sanity test script to import mllib and ml sub-components. Author: Holden Karau <holden@us.ibm.com> Closesapache#16465 from holdenk/SPARK-19064-fix-pip-install-sub-components.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment
5 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
Fix instalation of mllib and ml sub components, and more eagerly cleanup cache files during test script & make-distribution.
How was this patch tested?
Updated sanity test script to import mllib and ml sub-components.