Skip to content

[SPARK-18080][ML][PySpark] Locality Sensitive Hashing (LSH) Python API. - #15768

Closed
yanboliang wants to merge 2 commits into
apache:masterfrom
yanboliang:spark-18080
Closed

[SPARK-18080][ML][PySpark] Locality Sensitive Hashing (LSH) Python API.#15768
yanboliang wants to merge 2 commits into
apache:masterfrom
yanboliang:spark-18080

Conversation

@yanboliang

@yanboliangyanboliang commented Nov 4, 2016

Copy link
Copy Markdown
Contributor

What changes were proposed in this pull request?

Add MinHash and RandomProjection Python API.

How was this patch tested?

Add doc tests.

@SparkQA

SparkQA commented Nov 4, 2016

Copy link
Copy Markdown

Test build #68135 has finished for PR 15768 at commit 85d22c3.

  • This patch fails Python style tests.
  • This patch merges cleanly.
  • This patch adds the following public classes (experimental):
    • class LSHParams(Params):
    • class LSHModel():
    • class MinHash(JavaEstimator, LSHParams, HasInputCol, HasOutputCol, HasSeed,
    • class MinHashModel(JavaModel, LSHModel, JavaMLReadable, JavaMLWritable):
    • class RandomProjection(JavaEstimator, LSHParams, HasInputCol, HasOutputCol, HasSeed,
    • class RandomProjectionModel(JavaModel, LSHModel, JavaMLReadable, JavaMLWritable):

@SparkQA

SparkQA commented Nov 4, 2016

Copy link
Copy Markdown

Test build #68136 has finished for PR 15768 at commit cdeca1c.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@yanboliang

yanboliang commented Nov 4, 2016

Copy link
Copy Markdown
ContributorAuthor

cc @jkbradley@sethah

@yanboliangyanboliang mentioned this pull request Nov 7, 2016
Closed
@sethah

sethah commented Nov 7, 2016

Copy link
Copy Markdown
Contributor

I began to review this, but got sidetracked with a lot of the details we are currently discussing on the original LSH PR.

@jkbradley

jkbradley commented Nov 28, 2016•
edited
Loading

Copy link
Copy Markdown
Member

This can now proceed since http://github.com/apache/spark/pull/15874 is ready to be merged. Sorry for the delay! This will need to slip to 2.2

@jkbradley

jkbradley commented Jan 4, 2017

Copy link
Copy Markdown
Member

Pinging on this: What's a reasonable ETA for updating the PR? Thanks @yanboliang !

@yanboliang

yanboliang commented Jan 4, 2017

Copy link
Copy Markdown
ContributorAuthor

@jkbradley I just came back from vacation, and will update this PR before the weekend. Thanks.

@SparkQA

SparkQA commented Jan 24, 2017

Copy link
Copy Markdown

Test build #3550 has finished for PR 15768 at commit cdeca1c.

  • This patch passes all tests.
  • This patch does not merge cleanly.
  • This patch adds no public classes.

@YunniYunni mentioned this pull request Jan 26, 2017
Closed
@jkbradley

jkbradley commented Jan 27, 2017

Copy link
Copy Markdown
Member

Btw, @yanboliang and @Yunni did you sync? I'm fine with the takeover, but don't want to stomp on toes. Both can be listed as authors when this gets merged. Should we close this issue with the other taking its place?

@yanboliang

yanboliang commented Jan 28, 2017

Copy link
Copy Markdown
ContributorAuthor

I'm OK to close this one and glad to help to review #16715 , but I think I will have time until next week.

@yanboliang
yanboliang deleted the spark-18080 branch January 28, 2017 15:41
ghost pushed a commit to dbtsai/spark that referenced this pull request Feb 16, 2017
Yun Ni@yanboliang
[SPARK-18080][ML][PYTHON] Python API & Examples for Locality Sensitiv…
08c1972
…e Hashing
## What changes were proposed in this pull request?
This pull request includes python API and examples for LSH. The API changes was based on yanboliang 's PR apache#15768 and resolved conflicts and API changes on the Scala API. The examples are consistent with Scala examples of MinHashLSH and BucketedRandomProjectionLSH.
## How was this patch tested?
API and examples are tested using spark-submit:
`bin/spark-submit examples/src/main/python/ml/min_hash_lsh.py`
`bin/spark-submit examples/src/main/python/ml/bucketed_random_projection_lsh.py`
User guide changes are generated and manually inspected:
`SKIP_API=1 jekyll build`
Author: Yun Ni <yunn@uber.com>
Author: Yanbo Liang <ybliang8@gmail.com>
Author: Yunni <Euler57721@gmail.com>
Closesapache#16715 from Yunni/spark-18080.
cmonkey pushed a commit to cmonkey/spark that referenced this pull request Feb 16, 2017
Yun Ni@cmonkey
[SPARK-18080][ML][PYTHON] Python API & Examples for Locality Sensitiv…
8877cf1
…e Hashing
## What changes were proposed in this pull request?
This pull request includes python API and examples for LSH. The API changes was based on yanboliang 's PR apache#15768 and resolved conflicts and API changes on the Scala API. The examples are consistent with Scala examples of MinHashLSH and BucketedRandomProjectionLSH.
## How was this patch tested?
API and examples are tested using spark-submit:
`bin/spark-submit examples/src/main/python/ml/min_hash_lsh.py`
`bin/spark-submit examples/src/main/python/ml/bucketed_random_projection_lsh.py`
User guide changes are generated and manually inspected:
`SKIP_API=1 jekyll build`
Author: Yun Ni <yunn@uber.com>
Author: Yanbo Liang <ybliang8@gmail.com>
Author: Yunni <Euler57721@gmail.com>
Closesapache#16715 from Yunni/spark-18080.
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@yanboliang@SparkQA@sethah@jkbradley